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ABSTRACT 


The objective of this research effort is to design and implement a 
laboratory and establish a research method focused on scientific evaluation of 
human factors considerations for military individual equipment under both 
laboratory and field conditions. This integrated approach for laboratory and field 
conditions is the first of its kind for military human factors research, enabling an 
unparalleled degree of scientific rigor in the collection of empirical human factors 
data. This effort includes: 1) a state-of-the-art usability laboratory designed 
specifically for quantitatively evaluating military individual equipment; 2) a 
rugged, embarkable, fully self-contained, portable usability laboratory for field 
research in military environments; 3) a codified manual for using the two main 
configurations (stationary and portable) of the usability laboratory, written for the 
beginning usability researcher; 4) a set of validated procedures for applying 
sound human factors principles, and traditional and non-parametric statistics to 
the specific problem of usability testing of military individual equipment; 5) a 
proof-of-concept practical application of the laboratory and procedures to a 
specific problem, namely the usability testing of ruggedized personal digital 
assistants (RPDAs) designed for United States Special Forces operations. 


V 



THIS PAGE INTENTIONALLY LEFT BLANK 


VI 



TABLE OF CONTENTS 


I. INTRODUCTION.1 

A. Thesis Organization.1 

B. Probiem Statement.1 

C. Literature Review.5 

D. Objectives.21 

E. Scope, Limitations, and Assumptions.21 

II. DESIGN AND IMPLEMENTATION OF THE OBSERVATION 

LABORATORY.23 

A. Background.23 

B. Technical Specifications.25 

C. Installation.25 

III. APPROACH FOR USABILITY TESTING.27 

A. Overview.27 

B. Selecting Participants.28 

C. Conducting Interviews and Focus Groups.28 

D. Administering Surveys.28 

E. Conducting a Functional Task Analysis And Cognitive 

Task Analysis.31 

F. Conducting an Observational Usability Analysis.31 

G. Analytical Strategy.38 

IV. CASE STUDY.41 

A. Introduction and Rationale.41 

B. Method.42 

C. Case Study Results.58 

D. Discussion of Case Study.71 

V. CONCLUSIONS AND RECOMMENDATIONS.77 

A. Conclusions.77 

B. Recommendations.77 

C. Follow-on Research.78 

APPENDIX A. GLOSSARY AND ACRONYMS.81 


VII 































APPENDIX B. APPLIED WARFIGHTER ERGONOMICS 

USABILITY LAB BASIC USER GUIDE.85 

APPENDIX C. APPLIED WARFIGHTER ERGONOMICS CENTER 

INFORMATIONAL MATERIALS.147 

1. Naval Postgraduate School Public Affairs Office Article.147 

2. Applied Warfighter Ergonomics Observation Laboratory 

Data Sheet.149 

APPENDIX D. POINTS OF CONTACT.151 

1. Naval Postgraduate School (NPS).151 

2. RPDA Manufacturers.153 

3. Sponsors and other agencies.153 

4. Laboratory Equipment Suppiiers.154 

5. Defense Language Institute (DLI) at the Presidio of 

Monterey (POM).155 

APPENDIX E. SURVEYS.157 

1. Technology Experience Survey.157 

2. Handheld Device Evaiuation.161 

APPENDIX F. USABILITY TEST SCENARIO.167 

1. Instructions and Setup.167 

2. Conventionai equipment segment (Z).169 

3. RPDA segment of scenario: RPDA-A.172 

4. RPDA segment of scenario: RPDA-B.175 

5. RPDA segment of scenario: RPDA-C.177 

6. RPDA segment of scenario: RPDA-D.181 

7. Tactical Scenario Map.185 

APPENDIX G. RPDA TRAINING.187 

1. RPDA-A Lesson Plan Brief.187 

2. RPDA-B Startup Instructions.194 

3. RPDA-C Lesson Plan Brief.195 

4. RPDA-D Lesson Plan Brief.199 

5. RPDA and Basic Land Navigation Practical Application.201 

APPENDIX H. INFORMATION FOR PARTICIPANTS.203 

1. Research Brief.203 
































2. Letter to request participants.205 

3. Participant Information Form.207 

4. Consent Forms.209 

5. MapstoNPS.213 

6. Participant ID Form and ID Cards.215 

APPENDIX I. INSTITUTIONAL REVIEW BOARD FORMS.217 

APPENDIX J. SOFTWARE AND HARDWARE CONFIGURATION.219 

1. Noldus "The Observer" Software.219 

2. X-Keys Macro Keyboard.220 

APPENDIX K. DATA.221 

1. Focus Group Notes.221 

2. Survey Data: Individual Comments.225 

3. Survey Data: Individual Numerical Responses.229 

4. Survey Data: Aggregate Summary of Responses.235 

5. Usability Test Data.245 

APPENDIX L. MORS-TISDALE PRESENTATION.249 

LIST OF REFERENCES.263 

BIBLIOGRAPHY.267 

INITIAL DISTRIBUTION LIST.271 


IX 






















THIS PAGE INTENTIONALLY LEFT BLANK 


X 



LIST OF FIGURES 


Figure 1. S-Shaped Curve of Adoption from Rogers, 2003, pp. 273. 6 

Figure 2. The IBM “Out-Of-Box Experience” - a physically context- 

aware design (Selker & Burleson, 2000). 12 

Figure 3. The Traditional Engineering and Science “Black-Box” system 

model.13 

Figure 4. Context is everything but the explicit input and output 

(adapted from Lieberman & Selker, 2000). 14 

Figure 5. The components of usability, and an idealized learning curve 

(Jordan, 1998).20 

Figure 6 . ANOVA Model for Usability Measures.40 

Figure 7. Participant and RPDA Split-Screen View.45 

Figure 8 . Mapskill Function (algebraically and S-Plus code).61 

Figure 9. Pairs plots of mapskill, Z 2 , Zs, and Z 4 .62 

Figure 10. Pairs plots of mapskill, t 2 , ts, and t 4 .63 

Figure 11. General ANOVA Model for Usability Testing.64 

Figure 12. ANOVA analysis of task 5. 66 

Figure 13. ANOVA analysis of task 5, adjusted for participant effect. 66 

Figure 14. Kruskal-Wallis tests for device effects.67 

Figure 15. S-Plus Code for Kruskal-Wallis Multiple Comparisons Tests. 68 

Figure 16. Tasks times as coded by Rater A and Rater B.70 

Figure 17. Formula for Cohen’s k .70 


XI 




















THIS PAGE INTENTIONALLY LEFT BLANK 


XII 



LIST OF TABLES 


Table 1. Steps in Performing a Usability Analysis.27 

Table 2. Checklist for successful survey administration.30 

Table 3. Characteristics of Laboratory and Field Research.32 

Table 4. Checklist for setting up a usability analysis in the NPS AWE 

Observation Laboratory.36 

Table 5. Research Hypothesis for Equipment Testing.40 

Table 6. Estimated Time Spent in Data Collection.43 

Table 7. Hypothesis Hi: Differences between RPDAs.46 

Table 8. Surveys and Related Paperwork.48 

Table 9. Basic Tactical Tasks Used in this Study.49 

Table 10. Tasks considered, but rejected from this study.50 

Table 11. Tasks for consideration in future studies.50 

Table 12. Hypothesis H 2 : Conventional vs. Alternate Means.51 

Table 13. Coded tactical tasks.51 

Table 14. Errors by Type.53 

Table 15. Tactical Postures.53 

Table 16. Battle Gear Worn in the Tactical Scenario.54 

Table 17. Measures of Task Success.55 

Table 1 8 . The S-Plus data frame rpda . 60 

Table 19. Confusion Matrix for Participant 0513, RPDA-D.69 






















THIS PAGE INTENTIONALLY LEFT BLANK 


XIV 



ACKNOWLEDGMENTS 


I gratefully acknowledge the tremendous time and resources that my 
thesis advisor, Dr. Nita Lewis Miller, dedicated to this research. My thesis second 
reader. Dr. Nick Dew, has been an invaluable resource throughout my time at the 
Naval Postgraduate School for keen insight and direction, and has been 
encouraging and timely in all matters. Both professors have set personal 
examples for me with their knowledge, professional guidance, and dedication. 
Professors Robert Koyak and Lyn Whitaker reviewed my statistics, offering 
valuable suggestions and guidance in coaxing a coherent analysis out of some 
very messy data. 

I also received much assistance during the acquisition of equipment and 
conduct of experiments from Melissa Takagi, who is both the invaluable research 
assistant to Dr. Miller, and my incredibly patient, understanding, and supportive 
wife. 

Dr. Dave Netzer, director of the Tactical Network Topology project and the 
direct link between our laboratory funding and our sponsors at the Rapid 
Equipping Force and SOCOM, has made this research possible in the most 
pragmatic sense. He and his assistant, Mariana Verett, have been instrumental 
in all aspects of actually getting the lab built. 

Naval Postgraduate School graduate Chris Manuel has contributed greatly 
to my knowledge of the Tactical Network Topology project and all the associated 
hardware. Fellow NPS student Captain Francisco Caceres, USMC, has selflessly 
dedicated many hours of his own time to providing technical assistance and 
training to myself and other human factors researchers with various equipment. 
NPS students 1LT Derek Read, USAF, and 1LT Matt Simpson, USAF, have 
dedicated many hours to the establishment and testing of the Applied Warfighter 
Ergonomics Center, and assisted with the training of test participants. Katie 
Buettner, a student intern studying cognitive science at the University of 


XV 



California, San Diego, was instrumental in the successful conduct of the case 
study. She assisted in configuring the Observer software, running and 
troubleshooting the test scenarios, operating video equipment, coding 
observations, and finally in presenting the results. 

SFC Swanson and SSG Gregory Bumann at the 229**^ Ml Bn, and SSgt 
Boissoneau at the Marine Corps Detachment at the Defense Language Institute, 
were consummate professionals in arranging for DLI student participation in this 
research. 

My dear wife Melissa and my sons Jonathan and Christopher have been 
supportive, patient, and encouraging throughout my studies at the Naval 
Postgraduate School. They are my inspiration and my motivation and I am 
incomparably fortunate to have them. 


XVI 



EXECUTIVE SUMMARY 


End users bear the burden of poor design, and persuading users to learn 
and use a system’s capabilities goes beyond mere functional effectiveness. 
There is a human factors gap in the Test and Evaluation process, and a need 
within the Department of Defense (DoD) for a laboratory and a reliable research 
methodology to perform scientifically rigorous human factors evaluations that 
goes beyond subjective observations and survey data. Rather than "next-bench" 
testing between technical experts, true usability testing should involve 
representative participants under realistic conditions. 

The objective of this research effort is to design and implement a 
laboratory and establish a research method focused on scientific evaluation of 
human factors considerations for military individual equipment under both 
laboratory and field conditions. This integrated approach will represent the first of 
its kind for academic and military human factors research, enabling an 
unparalleled degree of scientific rigor in the collection of empirical human factors 
data. This effort includes: 1) a state-of-the-art, well-tested, and professional 
usability laboratory designed specifically for quantitatively evaluating military 
individual equipment; 2) a rugged, embarkable, fully self-contained portable 
usability laboratory for field research in all military environments; 3) a codified 
manual for using the two main configurations (stationary and portable) of the 
usability laboratory, written for the beginning usability researcher; 4) a set of 
validated procedures for applying sound human factors principles, and traditional 
and non-parametric statistics to the specific problem of usability testing of military 
individual equipment; 5) a proof-of-concept practical application of the laboratory 
and procedures to a specific problem, namely the usability testing of ruggedized 
personal digital assistants (RPDAs) designed for United States Special Forces 
operations. 

In March of 2004, the Special Operations Command (SOCOM) requested 
a human factors evaluation of the RPDAs they have fielded in Iraq and 
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Afghanistan. At that time, the DoD did not have a facility equipped specifically to 
test military individual equipment in the manner needed for comprehensive 
software and hardware usability testing. Consequently, a behavioral observation 
laboratory, the Applied Warfighter Ergonomics (AWE) Center, was designed and 
constructed for this purpose. Since then, the AWE Center has generated serious 
interest from a number of different DoD and federal agencies. It will also serve as 
the basis for much future research at NPS. The equipment has been modified at 
NPS with battery packs so that a researcher can follow a user into the field, 
recording how users operate the equipment and the context with which it is 
employed. All the necessary equipment fits into three waterproof transit cases, so 
that an actual field study could be conducted to validate the results obtained from 
a controlled laboratory experiment. 

Digital videos recorded in the laboratory or in the field are manually coded 
into data files using the Noldus Observer software. The technical details of the 
process have been written into a Users Guide, which distills hundreds of pages 
of documentation and usability references into a focused, stand-alone tutorial. 
The developer of the software is using our lab as a model on their website. 

This thesis details an experimental design, data collection method, 
analytical procedure, lessons learned and recommendations for future 
researchers. Both qualitative and quantitative means are employed to provide 
human factors usability testing. 

To validate the laboratory and the procedure, a small case study 
comparing three different RPDAs was conducted. The case study was a proof-of- 
concept and was not expected to be conclusive. Soldiers and Marines at the 
Defense Language Institute were brought to NPS in small groups. Participants 
were given a pre-operation survey with 45 questions that attempt to capture their 
technology experience and military background. The participants went through a 
30-minute tactical scenario, during which they performed various 
communications and navigation tasks while equipped with body armor and rifles. 
The scenario was scripted to drive the participants to perform the tactical tasks 



using conventional means (such as with a paper map), in order to provide a 
baseline measure of their abilities. Later in the scenario, they performed similar 
tasks using the RPDA. Quantitative measurements such as task time, tactical 
posture, error rate, and category of errors were recorded while users employed 
both conventional and RPDA means to perform the tasks. When quantitative 
results were not conclusive, recommendations were possible based on 
qualitative observations in the lab and the field. 

Inter-rater reliability statistics verified that the behavioral measurements 
were reliably coded using well-defined criteria. We compare the task times 
assigned by different researchers on the same video observation file and 
compute a confusion matrix, index of concordance, and Cohen's k. 

Quantifying individual differences between participants reduces the noise 
in the measurements, and reduces our required sample size. Testable measures 
of participant effect, like the baseline tasks, are preferable to surveys, which are 
prone to a variety of errors. With a properly constructed experiment, we can 
justify quantitative analysis of human factors data. 

Parametric and non-parametric statistics were used to compare devices in 
the case study. High participant attrition from the study limited the extent to which 
rigorous statistical analysis could be performed on the data. Some users voiced 
strong opinions about the devices in their post-operation surveys. This research 
has shown that perception can influence the adoption of innovations more 
strongly than quantitative performance. 

This thesis fills the human factors gap in the T&E process. There are 
Special Forces operators and Marines in Iraq and Afghanistan using all of these 
devices today. Results from this work and from research using this laboratory will 
be applied by SQCQM to improve current and future RPDAs. This laboratory will 
make the Naval Postgraduate School the premiere military individual equipment 
testing site in the country. 
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I. INTRODUCTION 


A. THESIS ORGANIZATION 

This research effort began as a human factors study of military ruggedized 
personal digital assistants within the Naval Postgraduate School's Tactical 
Network Topology project for the United States Army Special Operations 
Command. This thesis is organized into five major parts. After the introductory 
material, Part II: Design and Implementation describes the technical details of the 
laboratory designed and constructed to facilitate the study. Part III; Approach 
describes a general approach specifically for conducting a usability analysis of 
military individual equipment The case study reported in Part IV serves as a 
proof-of-concept for the laboratory and the approach, and is a complete study in 
its own right. The final conclusions and recommendations, Part V, discuss the 
future of this laboratory, its applicability to other problems, and ways to build on 
this groundwork to facilitate rapid, effective usability research of other military 
individual equipment. The appendices contain the data collected during the case 
study and other relevant resources and can be used as a guide for future data 
collection efforts. 

B. PROBLEM STATEMENT 

1. Description 

The successful development of an innovation from concept to an 
established product is closely linked to the user's ability and desire to employ the 
product. Developers who are too focused on the underlying technical capabilities 
of their inventions may be surprised that their products are not adopted by their 
targeted users. Apple Computer Fellow Alan Kay tells us that to users, “the 
interface is the application,” and features that are not apparent, do not exist. 

Traditional human factors research, including usability analysis, often 
relies on subjective data, such as surveys and free-form user feedback. These 

data can be difficult to reproduce and be prone to errors due to user bias and 
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distorted recollection. Military equipment places even greater demands on the 
usability analyst. Military equipment is rarely designed for, or even appropriately 
evaluated in, a controlled laboratory environment. Both the physical and 
psychological environments in which military equipment is employed have 
impacts on the human factors considerations of that equipment. 

Military individual equipment is that equipment that is normally carried and 
employed by a single person in combat. Examples include: personal weapons, 
body armor and protective gear, load bearing equipment, navigation aids, and 
portable communications equipment. Individual equipment is relatively 
inexpensive on a per-item basis. Unlike the research and development programs 
for larger and more expensive military items, individual equipment items may not 
be subjected to extensive human factors testing or systems integration with the 
“whole system” encompassing everything a ground combat Soldier or Marine 
may typically carry with them into battle. Equipment deficiencies and 
incompatibilities may not be noted until the equipment is actually fielded. 

The main challenges to performing usability analysis on military individual 
equipment are in apparent diametric opposition: 

• To be scientifically valid, the evaluation should be 
repeatable, comparable between systems, and ideally, yield 
statistically rigorous results. 

• To have predictive and inferential relevance, the equipment 
should be tested by representative users under realistic 
conditions. 

The demand for validity might lead us to perform a controlled laboratory 
study using convenient participants, such as students. The compromises in 
realism to capture accurate data and achieve statistically supportable results may 
lead to a test so artificial as to be pointless. Pursuing the goal of relevance, 
perhaps by conducting a field exercise using real Soldiers and Marines in a 
tactical scenario, results in a test that is not repeatable, with many variables 
being uncontrolled, confounded, and even unidentified. Highly subjective after¬ 
action reviews with operators are subject to unintentionally distorted recollection 


2 



and individual biases. Such data are also difficult to compare across different 
training scenarios 

Military individual equipment that is usable and appealing to users will 
increase the rates of adoption and effective employment. This thesis proposes a 
method for scientifically evaluating the usability and other human factors 
considerations of modern military individual equipment under both laboratory and 
field conditions. 

2. Background 

Various agencies within the Department of Defense are currently 
developing and fielding a number of handheld computing devices known 
collectively as Ruggedized Personal Digital Assistants (RPDAs). These devices 
are similar to, and often based on, the commercial personal digital assistants 
(PDAs) known as "Palm Pilots", "Pocket PCs", or "Windows CE" devices. Civilian 
PDAs, introduced by Apple in 1993 and popularized by Palm in 1996, introduced 
a new paradigm in computing and mobile information accessibility, and new 
human interface challenges to designers. Military handheld computers have 
unique human factors considerations. Civilian PDA interfaces are not designed to 
be used while the operator: 1) simultaneously attends to mental and physical 
external stimuli such as enemy activity, 2) wears and employs other equipment 
and weapons which may also require the use of both hands and eyes, or 3) 
operates under environmental extremes, such as wearing cold-weather gloves. 

As with any new technology, there is the challenge of encouraging and 
facilitating its adoption. The "diffusion of innovation" of electronic devices that 
supplant existing (manual) systems or provide novel capabilities can be slow 
when users have little experience with such devices, have low confidence in the 
electronic device, or have a mission-critical or life-and-death requirement for 
reliability. 

Decades of human factors research have produced guidelines for cockpit 
design, control panel layouts, and websites, among other interfaces. Currently, 
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there is not a well-defined methodology to evaluate the human factors issues or 
adoption strategies pertinent to military RPDAs. 

3. Rationale for Conducting Study 

Military RPDAs are at the cutting edge of human-machine interface 
technology. Human factors considerations are currently being accounted for in an 
ad-hoc and "gut-instinct" manner, if at all. This study will produce a rigorous 
method for examining the human factors considerations of military RPDAs. The 
results of this study will directly influence the ongoing development and fielding 
procedures of Special Operation Command's current military RPDA. Other 
organizations will be able to use this research to better design and field military 
RPDAs now and in the future. 

4. Research Questions 

What human factors considerations are generic to handheld devices under 
normal civilian operating conditions? 

What human factors considerations are specific to handheld computing 
equipment used by military personnel? 

How can these human factors issues be tested and evaluated in a 
laboratory environment, and what are the drawbacks of laboratory testing and 
evaluation? 

How can these human factors issues be tested and evaluated in a field 
environment, and what are the drawbacks of field testing and evaluation? 

What measures of performance and measures of effectiveness should be 
applied to the human interface of handheld military electronic devices? 

What factors do military personnel consider during the adoption of 
handheld electronic devices? How can a fielding plan address these factors in 
order to facilitate the successful training and employment of these devices? 
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What diffusion of innovation considerations should be incorporated into 
the research and development of military personal electronic devices? 

C. LITERATURE REVIEW 

Several fields of research are most pertinent to this study: human- 
computer interaction, usability testing, and the study of the diffusion of 
innovations. 

1. The Diffusion of Innovations 

Everett Rogers's Diffusion of Innovations (2003), in its fifth edition since 
the original was published in 1962, is widely recognized as the foundation work in 
the field. His basic approach and specific definitions are cited in virtually all 
diffusion and adoption literature. Rogers (2003) defines diffusion: 

The main elements in the diffusion of new ideas are: (1) an 

innovation (2) that is communicated through certain channels (3) 

over time (4) among the members of a social system. 

In a number of engaging case studies, he illustrates the ranges and 
implications of innovations, communication channels, time, and social systems 
within his framework. Rogers identifies five categories of adopters as having 
distinct characteristics and motivations with respect to innovation: innovators, 
early adopters, the early majority, the skeptical late majority, and the traditionalist 
laggards. He uses these and other specific generalizations to draw parallels 
between case studies of subjects as diverse as chicken farmers in Africa, doctors 
in Illinois, and cell phone users in Finland. The first edition of Diffusion of 
Innovations (Rogers, 1962) introduced the classic "S-shaped curve of adoption" 
(Figure 1). Through a series of extensive and rigorous experiments, Rogers 
demonstrated that the cumulative percentage of individual adopters of 
innovations (ideas, new technologies, or social practices) follows a normal 
probability distribution under a wide variety of circumstances. Rogers then 
demonstrates that the cumulative distribution of adopters generally follows the 
same S-shaped curve for organizations adopting innovations, as well. 
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Subsequent studies have established that the top official, head of household, or 
other nominal leader of the organization is not necessarily the decision maker in 
the adoption process. Identifying a specific individual responsible for adoption 
decisions is often complex or even fruitless. 


Figure 7-1. The Number of New .Adopters Each YeiU, iuid tlie 
Cumulative Number of Adopters, of Hybrid Seed Com iu Two Iowa 
Coiniminities 



Year 

The cumulative number of adopters of hybrid seed com approaches an 
S-shaped curve over time, while the Irequency distribution oi tbe number 
of mean adopters per yeiu- approaclies a nomuJ, bell-sliaped curv e. Early 
in the diffusivni process, relatively few indivitluals adopt in each tim<* 
period. Gradiuilly, however, the rate of adopHon speeds up until ;ill (or 
almost all) members of tbe system adopt the innovation. 

Source: Based on Ryan and Gross (1943). 

Figure 1 . S-Shaped Curve of Adoption from Rogers, 2003, 

pp. 273 
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Denrell and March (2001) offer mathematical models of adoption and use 
computer simulation to demonstrate how experiential learning in individuals and 
organizations can produce a "hot stove effect." A cat that sits on a hot stove lid 
once will never sit one again - but will never sit down on a cold stove lid, either 
[from a parable by Mark Twain], Denrell and March (2001) demonstrate that 
alternatives to the status quo (innovations) often share these characteristics: 

• initial performance (among novices) is lower than the expert 
performance seen after competence develops with the innovation, 

• initial performance has higher variation than later performance - 
systems are more stable at their final potential, 

• variability and risk of failure exist among alternatives, and final 
performance may be substantially less than the status quo. 

Denrell and March (2001) demonstrate through simulation that 
organizations that tend to persist in alternatives to reach a higher level of 
competence will therefore often persist in evaluating poor alternatives, and 
thereby fail and be weeded out. Due to high variability, risky alternatives will have 
runs of poor returns, eliminating individuals and organizations that take on risky 
alternatives. These effects combine to produce a systematic bias in favor of 
reliability, manifested as apparently "risk-averse" survivors. The factors that 
affect the success of alternatives and of individuals who adopt them include: 

• learning time required to gain competence, 

• variability in initial performance within an alternative (how risky is a 
particular alternative?), 

• variability in potential performance among all alternatives (how 
much better are the good alternatives?), 

• aspirations of desired performance (how much does the decision 
maker really want or need to improve to survive?). 

The factors that influence buyers deciding between products are the same 
as those that influence adopters of other types of innovations. Much research 
exists to explore how successful marketers encourage the adoption of their 
brands by consumers. 
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In contrast to analyzing what makes adopters switch to alternatives, 
Carpenter and Nakamoto (1989 and 1994) examined how leading brands come 
to dominate markets for decades. They propose that pioneering alternatives 
(first-to-market innovations) have an advantage beyond their initial uniquely 
desirable attributes. Not only can pioneers modify their product towards user 
preferences, but they also have the ability to directly influence market 
preferences. Through experiments with actual products and participants. 
Carpenter and Nakamoto (1989) showed that successful pioneering products 
could significantly influence user preferences in new markets. It is then much 
more difficult for later competitors to differentiate themselves from the pioneer. 
The successful pioneer shifts users' desires and preferences towards the 
pioneer's greatest strengths. 

Carpenter, Nakamoto, & Kent (1990) identified ways that later alternatives 
can be made more attractive in a market dominated by pioneers. Shankar, 
Carpenter, and Krishnamurthi (1998) followed up that research with strategies for 
late movers to reshape a market and arrest the pioneering brand's dominance by 
identifying a specific, superior position away from the pioneer. 

User preferences for alternatives can be manipulated in other ways. 
Carpenter, Glazer, and Nakamoto (1994) extended Rogers' (2003) principle of 
relative advantage - that one alternative is somehow objectively better than 
another for the adopter. Carpenter et al. (1994) showed that when alternative 
products have no meaningful differentiation between them (such as shampoo 
and instant coffee), consumers preferences can be significantly swayed by the 
successful marketing of irrelevant attributes that have no impact on objective 
product performance. Their experiments showed that consumer valuation of 
alternatives is not properly modeled by a rational choice interpretation. They also 
showed that there is significant interaction between the cost of the alternative 
and the perceived value of the irrelevant alternative - that is, a premium price 
tends to reinforce the value of an irrelevant alternative. As a specific example. 
Carpenter, Glazer and Nakamoto (1994) offered irrelevant differences in CD 
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players, pasta types, and down jackets to participants in a controlled experiment. 
The strength of the effect, that irrelevant attributes were positively valued in 
premium priced products, surprised the researchers. 

2. Human-Computer Interaction 

Functional system designs with human interaction must: 1) accept and 
process user input, and 2) perform some functional task. This is true for all 
manner of interactive systems (mechanical, electronic, or social). Context-aware 
designs present a different interface to the user(s) depending on the immediate 
context of use. 

A context-aware design (compared to a context-unaware interface for the 
same system) can improve user efficiency while decreasing errors and training 
requirements. Contextual awareness is becoming increasingly a part of modern 
user interfaces, information processing, decision-support, and command and 
control systems for military applications. Selker and Burleson (2000) introduced a 
framework for discussing computer designs that can be extended to all 
interactive system models: 

• task models (Lieberman and Selker, 2000) are the acts that the 
user directly performs to exercise some functional capability of the 
system, 

• user models consist of task-relevant background information about 
the user, 

• system models are the capabilities of the [computer] system. 

Norman (2002) analyzed a number of mechanical systems in which task 

models are not well-defined for the user, user models are improper or ignored, 
and system models are misleading or unavailable to the user. 

Rich (1999) argued that, “for many interactive computer systems, the user 
community is sufficiently heterogeneous that a single model of a canonical user 
is inadequate.” She presents three techniques for building user models: 
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• identifying the vocabulary and concepts employed by the user, 

• gauging the responses with which the user seems satisfied, and 

• using stereotypes (a collection of user traits) to generate many 
facts from a few. 

People are most familiar with machine systems that produce direct, 
computational, and highly accurate results from quantitative input data. “Expert,” 
or knowledge-based, systems produce indirect conclusions (often, qualitative, 
and in a restricted field) that closely parallels the conclusions drawn by a human 
“expert” in that field. Experts (human or machine) are believed to perform better 
than “novices” (non-experts) in part because they use superior internal models of 
the system (Jonassen, 2003). 

In case studies. Rich (1999) followed the development of an expert 
system that uses “stereotypes” to individualize a user model. Cognitive research 
has shown that a problem’s representation to the user has a tremendous effect 
on a subject’s internal model of the problem (Jonassen, 2003; Norman, 1993). 
Naturalistic decision making (NDM) theory asserts that, “often the biggest 
problem facing decision makers is that they do not know which questions to ask” 
(Meso, Troutt, & Rudnicka, 2002). 

Context-aware design can help bring novices up to expert-level 
performance by properly presenting the contextually relevant data. A user can 
then construct an internal model that accurately reflects the situation (in other 
language, build “situational awareness”). A context-aware design might then 
suggest (or conversely, suppress) possible courses of action, rather than 
overwhelm the user (often termed, “information overload”). 

System models can adopt a “process-centered” or a “product-centered” 
view of the system components (Abecker, Bernardi, Hinkelmann, Kuhn, & Sintek, 
2000). A context-aware design should be able to gather data on context not 
explicitly given to it, such as: time, place, weather, user preferences, skills, 
knowledge, and experience, and the history of interaction. These definitions of 
context can also be extrapolated to states unfamiliar to the system: not just, 
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“what is likely in the neighborhood right now,” but, more generally, “what is 
typically in a neighborhood like this” [emphasis added] (Lieberman & Selker, 
2000 ). 

Developing context-aware designs. Context-aware designs, by 
definition, respond to external conditions. They have a degree of sophistication 
that is unusual for machines. Harter, Hopper, Steggles, Ward, and Webster 
(2002) go so far as to describe a particular context-aware platform as “sentient”. 
Colloquially, the term “smart” is often variously used to refer to a system that: 

• performs complex tasks with high “ease of use”, or 

• uses a knowledge base to produce conclusions not obvious to a 
non-expert, or 

• attempts to predict or infer user action (that is, at least rudimentarily 
context-aware). 

The first definition merely indicates the initial employment of any 
technological system. This level of human factors engineering is well appreciated 
by industry (Vredenburg, 1999 and 2003), but has nothing to do with contextual 
awareness. A marvel of complexity may produce incorrect output, but be 
(erroneously) believed because it bears a false “pedigree criteria” for credibility 
(Vedder, 2004). 

Knowledge-based systems, built on an complex system model, must also 
assess the context of the data presented to the system, and the user’s state to 
produce useful and reliable results. Suchman (1987, ch. 6-7) described in great 
detail a case study of an expert system that has an excellent knowledge base 
and internal system model, but does not adequately collect information on the 
user’s context. Because it is context-unaware, its comprehensive knowledge 
base is unavailable to the user, and the system is a failure. 

Perception can create reality. Persuading users to learn and use a 
system’s capabilities goes beyond mere functional effectiveness. Design and 
aesthetics can affect a user’s perception, belief system, and even task ability 
(Csikszentmihalyi, 1990). Picard’s continuing research (2004) in “affective 
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computing” has shown that a critical part of our ability to see and perceive is not 
logical, but emotional. Her Affective Computing Research Group at the MIT 
Media Laboratory is studying the nature of human emotions, how to sense and 
model them (essential parts of a complete user model), and experimenting with 
“synthesizing emotions” in machines to create affective interfaces and 
applications (Picard, 2004). “Seductive” experiences create an emotional bond 
with their audience (Khaslavsky & Shedroff, 1999). A seductive interface 
enhances the value to users, and has the goal of persuading users towards 
some aim. Khaslavsky and Shedroff described three basic steps towards 
seduction: enticement, relationship, fulfillment. Technologies now exist that are 
actively persuasive in their own right, such as interactive agents that steer users 
towards a company’s product (Berdichevsky & Neunschwander, 1999). 



Figure 2. The IBM “Out-Of-Box Experience” - a physically 
context-aware design (Selker & Burleson, 2000) 


Selker and Burleson (2000) described physical, mechanical and electronic 
context-aware designs. For example, the setup instructions for an IBM™ laptop 
computer are not presented as a (context-unaware) traditional manual (Figure 2) 
The instructions are on strips of paper, in and on various parts of the computer, 
so that the user is presented with new actions only when they are useful (one 
strip is on the outside, another becomes visible when the machine lid is raised, 
etc.) Norman (2002) called these context-based constraints “forcing functions” - 
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situations in which actions are constrained (the system correctly interprets the 
user’s context) so that failure at one stage prevents the next step from 
happening. Norman (2002) also identifies a number of other physical and 
mechanical designs that notably assist or confound the user by their degree of 
contextual awareness. Forcing functions can be misapplied, and thereby prevent 
an appropriate means of recovery. This will lead to users quickly discovering 
ways to circumvent the system and fool the system’s user model, and in doing 
so, destroy the system’s context (Suchman, 1987, ch. 7). If users absorb such 
actions into their task model for the system, both performance, and confidence in 
the system, will suffer. 


Figure 3. 




The Traditional Engineering and Science “Black-Box” 
system model 


Harter et al. (2002) describe, in great detail, the technical issues 
concerning a specific hardware and software platform that gains contextual 
awareness (and maintains an accurate user model) by sensing the user’s 
location in a building. Abecker et al. (2000) describe the task models dealt with 
by their KnowMore project. The KnowMore project deals with the problem of 
“information overload”. A vast amount of knowledge is available through 
information retrieval tools (search engines), but by contextually modeling user’s 
information needs, the system eliminates unrelated documents and presents the 
user only with applicable knowledge and compulsory rules. Lieberman & Selker 
(2000) point out that there is a trade-off between the desire for abstraction and 
the desire for context sensitivity 

Difficulties in Designing Context-Aware Systems. Context-based 
designs break out of the traditional engineering and science “black-box” model of 
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system design shown in Figure 3. Lieberman and Selker (2000) propose an 
expanded, context-aware, systems design model, which is further modified in 
Figure 4. The traditional component-wise black-box decomposition is useful 
because it abstracts a particular system function from the (much more complex) 
whole. Black boxes are often simpler to design, test, and implement. It is exactly 
this abstraction that reduces context sensitivity. 



Figure 4. Context is everything but the explicit input and output 
(adapted from Lieberman & Selker, 2000) 

Vedder (2004) discussed the issues of humans assessing information 
reliability (specifically, information on the internet), and in what ways humans fail 
to recognize reliability or unreliability. Highly accessible information, such as 
results from an internet search engine, is often mistakenly held to be reliable. 
The fact that information comes from the internet can give it a misleading 
“pedigree of reliability” (Vedder, 2004). Vedder (2004) cited no research in this 
area, but his conceptual discussion can be extended to a measuring instrument 
that displays more significant digits than are actually accurate, or the display of a 
so-called “smart” interface. Humans are often at the extremes in their skepticism 
of machine-displayed information - either too credulous, or too dubious. Good 
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designs should incorporate a measure of reliability of their output, so as not to 
lead the user into a false context (of accurate data). Information density and user 
comprehension can be vastly improved with an appropriate choice of display. 
Tufte (2001) illustrated a wide variety of ways (good and bad) to display 
quantitative, multi-variate data. 

Berdichevsky and Neunschwander (1999) set out ethical principles of 
persuasive design, with a “golden rule” that, “actively persuasive technology 
should never seek to persuade anyone of something they themselves would not 
consent to be persuaded to.” People are persuaded (by other people) based in 
part on “liking” and “authority” (Cialdini, 1993). Within appropriate contexts, 
machine interfaces can similarly be made seductive, affective, and authoritative, 
enabling them to persuade users towards (or against) certain actions. 

Displays that do not take operator context into account may be ignored or 
rejected as inaccurate. Besnard, Greathead, and Baxter (2004) described how 
flawed mental models (particularly, in a detailed study of the British Airlines 737 
crash in 1989 in Kegworth, U.K.) can be mistakenly reinforced by coincidental 
system responses. 

“People’s actions are intimately intertwined with artifacts of their work; 
their team member’s roles, responsibilities, and actions; and even their cultural 
and historical setting,” (Olson & Olson, 2003, p. 495). Olson and Olson (2003) 
charted out how the field of Human-Computer Interaction is transitioning from 
early research in individual cognitive processes and motor skills, to a knowledge- 
based understanding of the role of context and “domain expertise” in user 
interfaces. 

The costs of poor design. Hackos and Redish (1998) point out a number 
of costs incurred by products that are designed without integrated user and task 
analysis. Cutting corners in the design process can result in lifetime expenditures 
in: 

• lost productivity. 
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• increased training and re-training time, 

• maintenance costs, 

• user support costs, 

• field maintenance costs, 

• correcting mistakes, 

• health and safety problems due to improper use, 

• administrative and legal costs due to improper use, 

• employee frustration leading to low productivity and contributing to 
attrition, 

• and lost sales or poor adoption. 

User interface design conciusions. Context-aware designs seek to 
improve user efficiency while decreasing errors and training requirements. 
Contextual awareness and presentation are usually, but not always, more 
complicated to incorporate into designs. Basic human factors considerations 
dictate that design should follow the human’s point of view, rather than the 
convenience of technical design. Beyond that level, designing contextual 
awareness requires considering the system from the human’s current point of 
view. A good context-aware interface must provide access to all relevant system 
capabilities, while aiding the user in making appropriate choices and/or restricting 
the user’s ability to make inappropriate choices. Poor attempts at context-aware 
interfaces may inadvertently restrict access to important capabilities, be jarringly 
inconsistent, or misinterpret the user’s actual context. 

The prevailing literature addresses the advantages of incorporating 
contextual awareness into user interfaces and whole-system designs. Actual 
examples of context-aware designs are primarily experimental, or in specific and 
novel applications. There exists little in the way of comparative studies of 
improvements gained by applying context-aware designs to existing, context- 
unaware interfaces. 
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Military information systems are often used in one or more of the following 
situations: 

• Decision makers must make sense of overwhelming amounts of 
relevant and timely data. 

• An operator or watch officer must have a formal “turnover” period to 
bring his or her relief abreast of the current situation (that is, 
establish their context of operations). 

• A user has considerable freedom to decide how to accomplish a 
specific task, and different users are more effective operating in 
their own, personalized “mode.” 

• Users have greatly varying degrees of skill and training, but all must 
operate the same system at a minimum proficiency. 

• A human, in the position of the system, would have implicit 
knowledge of many environmental factors (time, weather, historical 
context) that would not be explicitly entered into a system. 

These situations all are candidates for improvement through context- 
aware system design, and context-aware interfaces. Context-aware design 
applied to military individual equipment could be implemented in the following 
areas. 


Sensible default behavior. The most likely choice is presented to the 
user. For example, novices need not enter complicated networking configuration 
information if the system can determine some configuration that would work. 
Advanced users might optimize their settings in some other manner. 

Restricted options. For users of various categories, based on skill level, 
tactical role, or security requirements. 

Robustness to inputs. Users of tactical equipment will exit applications 
and turn off the device at any possible time. Components may be disconnected 
at any time during operation, and power sources may be intermittent. Devices 
should be robust to all such conditions without corrupting or deleting data or 
physical damage to equipment. 

Informative error conditions and graceful recovery. Error messages 
should recommend an appropriate recovery procedure (for example, a sensible 


17 



default behavior). A universal reset procedure should be available to restore a 
system to normal operation. 

Users should be drawn from typical current users ranging from trainees to 
highly experienced experts. The demographics and psychographics of the 
representative group should be compared with those of the larger population of 
actual users, and significant departures be noted. Pre-interviews with the 
subjects should identify the following: 

• preconceptions and biases about the system, 

• their existing understanding of the user, task, and system models, 
and 

• previous experience with similar systems. 

An aspect of contextual awareness that is not well discussed in the 
literature is the trade-off between better performance, using contextual 
awareness and the benefits of standardization across users. This has particular 
significance for users of military designs. For a number of reasons in training and 
wartime, personnel changes often happen rapidly in military units. Replacement 
personnel may have a wide range of skills and experiences. All military systems 
must be designed in such a way that users can be substituted for one another 
without warning, which has implications for how far context aware designs can 
customize themselves to a specific user model. 

A primary purpose of this research is not to develop the “how” of system 
design, but the “why” - why, and by how much, is a high technology solution 
better than a traditional system? In what areas does high technology strongly 
benefit the overall system, and in what areas are diminishing returns seen from 
applying high technology solutions? The answers to these questions will 
stimulate further research with at least two goals: 1) to improve specific existing 
systems, and 2) to encourage system designers to incorporate usability analysis 
in future system designs from their inception. As with other human factors 
improvements, it is far easier to create a system with the proper perspective in 
mind from the beginning, than to retrofit poor designs later. 
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3. Usability Testing 

In the standards document ISO 9241-11, the International 
Standards Organization (ISO) defines usability and three related 
terms: 

Usability: The effectiveness, efficiency, and satisfaction with which 
specified users can achieve specified goals in particular 
environments. 

Effectiveness: the extent to which a goal, or task, is achieved. 

Efficiency: the amount of effort required to accomplish the goal. 

Satisfaction: the level of comfort that the user feels when using a 
product and how acceptable the product is to users as a vehicle for 
achieving their goals. 

In his brief and very readable An Introduction to Usability (1998), Jordan 
identified and discusses the user characteristics that have the most direct effects 
on usability: experience, domain knowledge, and cultural background. Jordan 
(1998) also included disability, age and gender in his list, but within the narrower 
context of military usability studies, these characteristics are less likely to be 
relevant. 

There are various taxonomies of errors (see Wiegmann & Shappell, 2003, 
Reason, 1990, Dekker, 2002, and Norman, 2002). The errors commonly 
measured in usability testing include slips and mistakes. A slip occurs when the 
user knows how to perform a task, but unintentionally does something wrong 
during the task. A mistake occurs when the user has an erroneous model of how 
the product works (Reason, 1990). Slips and mistakes may indicate different 
reasons for the commission of an error, but may be difficult to distinguish in some 
contexts. 

As illustrated in Figure 5, Jordan (1998) defines five components of 
usability in a manner pertinent to the analysis of military systems (boldface and 
italics added): 
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Guessability. The effectiveness, efficiency, and satisfaction with which 
specified users can complete specified tasks with a particular product for the first 
time. 

Learnability. The effectiveness, efficiency, and satisfaction with which 
specified users can achieve a competent level of performance on specified tasks 
with a product, having aiready compieted those tasks once previously. 

Experienced User Performance (EUP). The effectiveness, efficiency, 
and satisfaction with which specified experienced users can achieve specified 
tasks with a particular product. 

System Potential. The optimum level of effectiveness, efficiency, and 
satisfaction with which it would be possible to complete specified tasks with a 
product. 


Re-usability. The effectiveness, efficiency, and satisfaction with which 
specified users can complete specified tasks with a particular product after a 
comparatively long period away from those tasks. 



EXPERIENCE WrTH PRODUCT 

Figure 5. The components of usability, and an idealized 
learning curve (Jordan, 1998). 

Why not simply ask users or subject matter experts (SMEs) what they 
want? Hackos and Redish (1998) point out that users themselves do not know 
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how to articulate what they do, especially if they are very familiar with the tasks 
the perform. Users will tend to emphasize difficult, boring, or exciting tasks. 


D. OBJECTIVES 

The objective of this research effort is to design and establish a laboratory 
and research method specifically focused on scientifically evaluating the human 
factors considerations of military individual equipment under both laboratory and 
field conditions. This laboratory will represent the first of its kind for academic and 
military human factors research. The specific results of this effort include: 

• a complete, well-tested, and professional usability laboratory 

designed specifically for evaluating military individual equipment, 

• a rugged, embarkable, fully self-contained portable usability 

laboratory for field research, 

• a codified manual for using the two main configurations (stationary 
and portable) of the usability laboratory, written for the beginning 
usability researcher, 

• a tested set of procedures for applying sound human factors 
principles to the specific problem of usability testing of military 
individual equipment, 

• a proof-of-concept practical application of the laboratory and 

procedures to a specific problem, namely the usability testing of 
RPDAs designed for ground combat personnel. 

E. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

This research effort was limited primarily by time and the availability of 
appropriate participants. In order to both design and fully exercise the capabilities 
of the laboratory, this thesis was intended to be limited as follows. 

Review the extant literature pertinent to the human factors of RPDAs. 
Identify military RPDAs currently under development. 

Discover the requirements for, design, construct, and create training for a 
usability laboratory suitable for testing and evaluating handheld electronic 
devices. 
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Define a methodology for evaluating the human factors considerations of 
military PDAs, including the software and hardware of behavioral observation 
systems, data collection procedures, and training. 
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II. DESIGN AND IMPLEMENTATION OF 
THE OBSERVATION LABORATORY 

A. BACKGROUND 

The bulk of the time in this effort was spent in designing, constructing, 
learning about, and experimenting with the equipment used for the Applied 
Warfighter Ergonomics (AWE) Observation Laboratory. The technical details of 
the installation and use of the lab are given in step-by-step detail in the Applied 
Warfighter Ergonomics Observation Laboratory Basic User Guide (Appendix B, 
also published as a stand-alone document). 

During the pilot experiments leading up to the case study and during the 
case study itself, students were brought in to use the equipment in the AWE 
Observation LabL These students exercised the various capabilities of the lab, 
including capturing and editing video, creating DVDs, creating configurations in 
The Observer software, and coding behavior in The Observer software. They 
referred only to the Basic User Guide created during this project (found in 
Appendix B), with occasional coaching where necessary. Feedback during this 
process refined the Basic User Guide to the point where the last users required 
no coaching beyond the tutorial in the Basic User Guide to fully perform a basic 
usability study. 

The Naval Postgraduate School Applied Warfighter Ergonomics 
Observation Laboratory (NPS AWE Observation Lab) was designed to be the 
first observation laboratory designed explicitly to investigate military individual 
equipment. Traditional observation laboratories exist in the civilian community 
that are used variously to study computer interfaces, websites, and small 
physical devices (Noldus, 2005). There already exist many (often conflicting) 
human interface guidelines for software, as well as detailed ergonomic studies of 


1 Students who used the AWE Observation Laboratory during its initiai testing included: NPS 
Human Systems integration students Petra Aifred, Tim McKneiiy, Derek Read, and Matt 
Simpson, and UCSD undergraduate cognitive science student Katie Buettner. 
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highly sophisticated, expensive military equipment such as aircraft cockpits and 
nuclear submarine control stations. Currently, no usability lab exists which is 
dedicated to the study of the equipment carried by a single soldier or Marine, and 
the behavior and characteristics of that ground combat Warfighter. 

The AWE Observation Laboratory was conceived with this mission 
statement: 

To design and establish a laboratory and a methodology for 
scientifically evaluating the human factors considerations of combat 
equipment used by military personnel under both laboratory 
controlled and realistic field conditions. 

The state of the art in behavioral observation hardware and software was 
sought, in conjunction with the appropriate military equipment to focus the lab on 
military individual equipment. The following is a general list of the functions that 
were required to establish this laboratory as uniquely serving the ground 
combatant: 

• robotic digital cameras to record behavior from multiple angles, 

• video mixing and editing hardware and software, 

• standard issue military individual equipment, such as helmets, body 
armor, packs, and training rifles, 

• large screen plasma display for presenting tactical scenarios to 
participants, 

• wired and wireless network access to the Naval Postgraduate 
School research network, 

• GPS rebroadcasting equipment allowing the reception of real GPS 
signal inside the laboratory (where GPS signals are usually 
unavailable), 

• wired and wireless network access to the NPS “Gigalab” Tactical 
Network Topology experimental research network so that the latest 
networked computing devices developed at NPS could operate 
within the lab. 

Establishing these capabilities required close coordination with the 
following individuals and agencies. 
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• Dr. Dave Netzer’s Tactical Network Topology (TNT) research 
program, including a number of students involved in the quarterly 
TNT experiments, 

• Dr. Alex Bordetsky’s Gigalab research group, 

• Captain Francisco Caceres, student in the NFS Information 
Technology Management curriculum, for his work with RPDAs, 

• the Marine Corps Tactical Systems Support Activity in San Diego, 
California, 

• Noldus Information Technology of Leesburg, Virginia, developer of 
The Observer software and related observation hardware, 

• NFS Public Works department for the installation of a two-way 
mirror, a GPS antenna on the roof of Glasgow, telephone systems, 
and other small construction projects, 

• NPS Information and Technology Assistance Center (ITACS) for 
the assignment of static IP addresses to various equipment and 
network connectivity to the NPS domain for the computer systems, 

• Army and Marine Corps staff at the Defense Language Institute 
(DLI) at the Presidio of Monterey (POM), for access to DLI students 
as experimental participants. 

Detailed point of contact information for the individuals and agencies listed 

above can be found in Appendix D. 

B. TECHNICAL SPECIFICATIONS 

A detailed list of all the equipment, by manufacturer and model number, 
and complete wiring diagrams can be found in the Applied Warfighter 
Ergonomics Observation Laboratory Basic User Guide (Appendix B). The user 
guide also contains a complete step-by-step tutorial that would take an 
inexperienced user through the laboratory hardware and software. An 
accompanying DVD shows the equipment and procedures for using the 
laboratory. 


C. INSTALLATION 

Initial installation of the Observer computer system was performed in a 
single day by Koichi Takagi of NPS, and Bart von Roekel and John McGraw of 
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Noldus Information Technology. In addition to the observation equipment 
purchased from Noldus, the basic requirements for the observation lab included: 

• two adjacent rooms (the “participant” and “observer” room) 
measuring at least 10’ x 10’. A much larger room is preferred for the 
participant room. Each room should be able to be fully isolated from 
other areas, and have comfortable lighting and temperature 
controls. Each room must have both electrical and computer 
network outlets. The observer room may require as many as twenty 
surge-protected electrical outlets for the observation equipment and 
supporting various equipment for testing (battery chargers, radios, 
etc.) 

• access between the rooms (for example, through a suspended 
ceiling) for a considerable amount of cabling, 

• wall-mounted remote cameras and their associated cabling, 

• installation of a piece of two-way mirrored glass between the 
rooms. This step required extensive coordination with local facilities 
maintenance. The glass was installed with the lower edge 
approximately 36 inches above the floor, so that a seated observer 
would have full view of the participant over his or her desk. The 
observer room must be darkened to fully prevent the participant 
from seeing the observer through the glass. Blinds were installed in 
front of the two-way mirror, and shelves installed near the cameras, 
to reduce the participant’s self-consciousness and awareness of 
the observer. 

The observation equipment requires ample shelf and desk space for 
computers, multiple video monitors, the camera controller and video mixer, note¬ 
taking materials, and other equipment. Additional adjustments and improvements 
to the laboratory space required approximately forty additional hours spread over 
many months. 
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III. APPROACH FOR USABILITY TESTING 


A. OVERVIEW 

This thesis is intended to dramatically reduce the demands of performing 
effective usability research on military individual equipment by providing: 1) 
equipment customized to the task, 2) a validated, step-by-step procedure for 
performing all phases of the research from data collection to analysis, and 3) a 
template (the case study in part IV) for presenting such a study. 

Table 1 lists the steps in performing a usability analysis of military 
individual equipment using the Applied Warfighter Ergonomics Center 
Observation Laboratory. The rest of this section will describe each step in further 
detail. 


Table 1. Steps in Performing a Usability Analysis 

1. Concept 

2. Definition of need 

3. Design of lab study 

4. User identification 

5. Focus groups/interviews 

6. Pre-surveys 

7. Lab testing - obtain quantitative performance metrics 

8. Post-operation survey (operation in lab) - qualitative user 

impressions 

9. Analysis 

10. Design of field study 

11. Field testing - validation of laboratory results through 
quantitative metrics 

12. Post-operation survey (operation in the field) - qualitative 
user impressions 

13. Report 


27 




B. SELECTING PARTICIPANTS 

Participants in any usability study should represent a sample of the 
intended user population. An irrelevant test population is a threat to external 
validity, to generalizability, and to credibility with the military consumers of the 
final report. Assumptions about true user populations and test user populations 
should be determined, clearly stated, and tested where possible to maintain the 
validity of the study. 

A qualitative profile of the typical user population can be developed 
through focus groups, and interviews with users, subject matter experts, and their 
leadership, and surveys. Test scores and quantitative ergonomic data, such as 
physical measurements and capabilities, can also be used to define a working 
“stereotype” of the user population (Rich, 1999). 

C. CONDUCTING INTERVIEWS AND FOCUS GROUPS 

Interviews and focus groups are a method used to gain insight into issues 
concerning newly designed equipment. Participants may include: sponsors, 
users, developers and subject matter experts. The interviews and focus group 
discussions are used to determine things such as: user needs, assumptions and 
biases, past experiences of the stakeholders, and the conditions under which the 
devices may used. Focus groups, interviews, and casual conversations with the 
stakeholders may reveal issues that would not surface elsewhere. Cozby (2004) 
notes that the group interaction in a focus group can stimulate a variety of 
responses that might not surface in surveys, interviews, or testing. 

D. ADMINISTERING SURVEYS 

Bordens and Abbott (2002) and Cozby (2004) describe many of the issues 

around conducting effective survey research. In the surveys designed for this 

research, both open- and closed-ended questions were used to get as much 

information as possible from concise (3-5 page) forms. The limited availability of 

relevant test participants makes it important that the survey questions be clear 

and unambiguous, and not be overly time-consuming. As suggested by Bordens 
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and Abbott (2002), demographic items are not presented first on the 
questionnaire, and related items are presented together. For consistency and 
ease of scoring, a five-point Likert scale ("very dissatisfied" to "very satisfied") is 
often used to gauge qualitative responses. 

In most situations, criteria for grouping the individuals may not be 
apparent until after the survey data is collected, and the researcher must take 
care not to prematurely group responses into categories. A question such as, 
“How many hours per week do you play video games,” is better answered with a 
continuous response (e.g., “enter number of hours”) than with pre-assigned 
categories (e.g., “0-1 hours, 1-3 hours, more than 3 hours”) to avoid ceiling and 
floor effects. After the data are collected, the responses can be grouped into low, 
medium, and high levels, if desired. Note that some types of statistical analysis 
are more appropriate with continuous values (the actual response in hours) than 
with categorical (low, medium, high). However, there may be reason to suspect 
that such survey data, based on estimates and individual recollection, is not 
really providing the numerical accuracy implied in the response. In that case, 
grouping responses into categorical levels may be more accurate. 
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Table 2. Checklist for successful survey administration 

1. Initial concept of survey - goals, participants, timeframe 

2. Institutional Review Board (IRB) approval) 

3. Review survey internally (peers and faculty) 

4. Administer pilot survey - edit and refine survey 

5. Administer actual survey 

6. Provide plenty of copies of all paperwork 

7. Provide pencils or pens, and clipboards or a writing surface if 
necessary 

8. Introduction and instruction letters to participants 

9. Allow adequate time to complete survey 

10. Prepare folders and portable storage (filing box, etc.) for copies 

of blank forms and surveys, and for completed forms and 
surveys (see the example in Table 8). _ 


1. Pre-operation Surveys 

Users should complete a survey before using the equipment to assess 
their demographic background, relevant experience and skills, and potential 
biases. The survey responses can be aggregated with the goal of identifying 
similar groups of users, such as technical novices and experts, or early adopters 
and hold-out traditionalists. Analysis of the responses may also uncover specific 
independent variables (IVs) that are indicators or influences on final 
performance. 

2. Post-operation Surveys 

Users should complete a survey after using the equipment to obtain 
qualitative feedback from the user on the equipment. The results of this survey 
can be used in conjunction with the usability testing to determine how well users' 
impressions of their performance and satisfaction with the equipment 
corresponds to their measured performance. As with the pre-operation survey, 
the responses can be used to identify categories of users, by their self-reported 
success and satisfaction measures. Of special interest would be areas where 
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users self-reports (e.g., survey or verbal feedback) disagreed with their 
measured performance. For example, a user expressing a high level of 
dissatisfaction with a device might exhibit a higher level of performance on that 
device than on a device with which she was more satisfied. 

E. CONDUCTING A FUNCTIONAL TASK ANALYSIS AND COGNITIVE 

TASK ANALYSIS 

Tasks performed during testing should be as closely related as possible to 
the real-world tasks of the user. These tasks can come from a formal operational 
requirements document (ORD), from less formal statements gathered from focus 
groups and interviews, or empirically, from a task analysis conducted as a part of 
the research effort. A functional task analysis may also be used to validate and 
refine in detail the tasks defined in the ORD or from other sources. 

The functional task analysis of how the equipment tested is actually used 
will identify the tasks that are most relevant to the equipment’s operation, and the 
tasks that are practical to test quantitatively. The task analysis may also clarify 
the allocation of functions within a larger context. Some tasks are best performed 
by a human, some are better performed on a device,: these divisions may be 
situationally dependent. 

Hackos and Redish (1998) and Kirwan and Ainsworth (1992) detail many 
of the various techniques for conducting task analyses, including verbal 
protocols, observations, and task decomposition. Rubin (1994) and Cozby 
(22004) present concise and effective recommendations for conducting surveys, 
usability testing, and experimental design. Their texts were used extensively in all 
phases of this research. 

F. CONDUCTING AN OBSERVATIONAL USABILITY ANALYSIS 

The process of turning human behavior and observed events into a 

quantitative data set is called coding or scoring. Coding can involve numerical 

data such as recording times and counting behaviors. Behavior can also be 

coded in categorical terms, such as types of behaviors, or be more subjective, 
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such as gauging emotions or intent. The goal of such an analysis is a written 
record that accurately characterizes the experimental session. 

A usability test should define tasks to be evaluated, the precise conditions 
under which they will be performed, and specific standards that quantify success 
or otherwise measure performance. Usability data can be collected in the 
laboratory or in the field. Table 3 lists the important characteristics of laboratory 
and field research in the context of usability research. 


Table 3. Characteristics of Laboratory and Field Research 


Laboratory research 

Field research 

Controlled conditions 

Uncontrolled conditions 

Repeatable scenarios 

Unrepeatable scenarios 

Artificial 

Realistic 

Can collect a large amount of highly 

Reveals data not seen in lab 

detailed data 

Expensive in time and other resources 
More errors in data collection 

Many confounds 

Allows statistically rigorous conclusions 

Validation of laboratory results 


1. Tasks 

Tasks to be evaluated are legitimate when they are defined in a 
supportable manner and validated through an independent functional task 
analysis. To the greatest degree possible, tasks should reflect real-world 
employment of the equipment tested within the artificial conditions of a 
laboratory, safety and practical restrictions in a field exercise, and the limits of 
direct measurability. When performing a comparative analysis, tasks should be 
limited to those that can be performed similarly in each testing condition, whether 
those be different devices or different groups of users. 

Within the Observer software, each task is defined as a separate 
behavioral class containing two state behaviors: ON task and OFF task. When 
defined as state behaviors, the software records the period that the task is 
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performed as a time interval (rather than just as two discrete points in time), 
which is useful for later analysis. 

2. Conditions 

Precisely defined, rigorously controlled experimental conditions ensure 
that the research has sufficient control to maintain internal validity, that the 
experiments can be replicated, and the data can be appropriately analyzed. 
Controlling situations involving human behavior can be more complicated that it 
seems, even within the structure of a laboratory experiment. Conditions should 
be recorded to the level of detail that they can be replicated by future researchers 
with no further details provided. At a minimum, researchers must write a detailed, 
scripted scenario, that includes: 

• a timeline, 

• a description of what each researcher and participant will be doing 
at each point in the scenario, 

• detailed notes on test equipment settings, 

• detailed notes on environmental conditions, such as lighting, and 
time of day, 

• indications of what will trigger each event or behavior of interest, 
that is, how does the participant know to start a task? 

• what equipment will be worn or used, such as helmet, rifle, or 
handheld device. 

The conditions under which the experiment is performed should be 
recorded as independent variables (IVs) for the session and possibly used in 
later analysis. 

3. Standards: Measures of Effectiveness and Measures of 
Performance 

Tasks and behaviors within a usability test should be evaluated using 
unambiguous, precisely defined standards coded by independent researchers. 
Common usability measures are: speed and accuracy (task time), and accuracy 
(# of errors by type). Other human factors metrics, which are more difficult to 
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measure, include: initial training time required, skill retention over time, and a 
device’s physical accessibility or ergonomics. 


Measures of effectiveness (MOEs), measures of performance (MOPs), 
and data requirements have precise definitions within military Test and 
Evaluation (Hoivik, 2005): 

MOE. A measure which expresses the extent to which a combat 
system accomplishes or supports a mission or task. MOEs are 
designed to address an issue, and should be relevant, complete, 
precisely defined, and quantifiably measured. MOEs should be 
directly traceable to a requirements document. 

MOP. A quantitative or qualitative measure of a system’s 
capabilities or specific performance function. MOPs are often 
expressed as distances, times, rates, or ratios 

Data Requirement. A number or other measurement which can be 
collected at a single location, does not depend upon recall or 
judgment by the data collector, and can be derived directly from the 
MOPs. 

4. Overview of a Usabiiity Test in the NPS AWE Observation 

Laboratory 

To use the AWE Center Observation Lab effectively, researchers should 
prepare the following: 

a. a well-formed research question to investigate. This question 
may emerge from a formal requirements document, or from a focus group or 
brainstorming session, 

b. a set of the proposed independent variables (IVs) that will be 
controlled in each observation session. This list may include information about 
the participant (name, rank, gender, etc.) and information about the scenario or 
environment ("wearing equipment load A", "night conditions", etc.), 

c. a list of tasks to be given to the participant (e.g., "send a 

message"). 
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d. a detailed list of "behaviors" (or "events") that will be observed 
and recorded. Virtually all possible behaviors of interest should be defined 
beforehand ("start task", "press button 1","user commits error"..., "complete 
task"). While not all events will occur in each scenario, some events may occur 
multiple times. 

e. a detailed, scripted scenario, as previously described, 

f. any required paperwork such as IRBs, consent forms, 
background questionnaires, etc. 

Researchers will produce video files of the observation session for each 
participant, and use Noldus Observer to score the behaviors observed. The data 
can be exported in standard formats (such as Microsoft Excel or comma- 
separated values), and analyzed with Noldus Observer’s built in statistics 
package or external statistics programs of choice. 

Multiple coders or researchers can score the same sessions from video 
files, and then conduct a reliability analysis between the coders (inter-coder 
reliability). The same coder can be compared against previous observations to 
measure intra-coder reliability. 

The main phases of using the observation laboratory are to set up the 
experiment, record the session to video, score the behaviors observed during the 
session, and analyze the data. Table 4 provides a checklist for successfully 
performing such an observation session. 

Practice runs before actually attempting to collect data should be 
considered absolutely necessary. 

Conducting experiments with real human participants and erratic 
equipment may introduce last-minute changes to the experimental design. When 
at all possible, contingencies (such as malfunctioning equipment and missing 
participants) should be planned for in advance, rather than reacted to under 
pressure. 
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Table 4. Checklist for setting up a usability analysis in the NPS AWE 

Observation Laboratory 

1. General 

1.1. Reserve the AWE Observation Lab well in advance. 

1.2. Confirm the date and time with all participants and others involved. 

1.3. Open HSIL main room (GL-221, phone 656-3138), participant (GL-221a), 
and observer (GL-216, phone 656-3134) rooms. 

1.3.1. Check lighting and inspect area for cleanliness and safety issues. 

1.3.2. Secure any equipment, papers, etc. that could interfere with your 
session or confuse participants. 

1.3.3. Put up “RECORDING IN PROGRESS” signs and inform others in 
the area of your work. 

1.4. Review the script or run profile(s) for the observation(s) to be made. 

1.5. Brief participant(s) on cameras, microphones and general lab procedures. 

1.6. Ensure consent forms are properly filled out and on file. 

2. Set up participant room (GL-221 a) 

2.1. Arrange furniture in accordance with script. Pay attention to camera 
locations and try to keep the participants’ normal movements from 
blocking the cameras. Tape marks on the floor to indicate where 
participants should stand. 

2.2. Have an assistant stand on the marked spot while you preset the camera 
positions. 

2.3. Provide paper, pencils, and other props (as required). 

2.4. If used, turn on computer (power strip, CPU, display). 

2.5. Participant workstation operations check (as applicable). 

3. Set up observer room (GL-216) 

3.1. Have script, notepad, and pens available for taking notes during the 
session. 

3.2. Turn on power to three surge protectors, power button on computer. 

3.3. Visually check that power is on to: surge protectors, computer, computer 
speakers, printer, event logging keyboard, two amps, VCR, two monitors, 
camera controller, and video mixer. 

3.4. Log on to the Observer workstation as Noldus Observer (no password 
required). 
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5. Challenges in Coordination 

A usability experiment, by nature, involves human test participants, 
unfamiliar tasks, and untested equipment. The researcher should always prepare 
for unforeseen and time-consuming delays in bringing these pieces together. 

When conducting any study involving human participants, researchers 
should do their utmost to incorporate flexibility around the participants in all 
phases of the study. The experiment should impose minimal demands on test 
participants. Their interest in participation (and suitability as participants) wanes 
quickly if they must wait excessively due to equipment malfunctions or 
scheduling conflicts. Participants may be unavailable during the planned testing 
period. 

While the usability researcher may be expert in running a usability test, the 
specific tasks used in the experiment may be wholly unfamiliar. Pilot studies are 
key in identifying the missing pieces of task descriptions that were left out during 
subject matter expert interviews. The usability researcher is not likely to become 
an expert in an unfamiliar domain during the experiment, but must have at hand 
precise and replicable definitions of all tasks, conditions, and standards. These 
should be reviewed extensively with the research sponsor or other domain expert 
to ensure that the experiment will actually observe and test the tasks and 
behaviors of interest. 

The researcher should not strive to become a technical expert on the 
product or device being evaluated. Many products tested will be unstable 
prototypes or highly technical in nature. A domain expert on the device should be 
on hand to train participants, setup and troubleshoot equipment, and review the 
test scenario for practicality. It is inadequate and unrealistic to expect that the 
researcher conducting usability assessments on a device will master it in a short 
time, teach it to participants, and set up and troubleshoot it during the 
experiment. The effort this requires will inevitably result in negative compromises 
in either the scope of usability test or the correct employment of the equipment, 

and most likely in both areas. Convincing sponsors and other agencies to provide 
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proper support for the equipment to be tested was a major challenge during this 
research effort, but should be vigorously pursued. During this research, the 
technical experts quickly removed roadblocks to success that would have been 
otherwise insurmountable. 

A domain expert may also advise the usability researcher on the event 
triggers and the range of subsequent behaviors likely to be observed. The 
usability researcher must assiduously maintain his or her impartiality during this 
relationship. The domain expert will often be an advocate or developer of the 
product being tested, and their motivations and biases must be recognized and 
acknowledged at every step of the experiment. Most developers are 
professionals and unlikely to intentionally or maliciously conceal negative aspects 
(or overly emphasize positive characteristics) of their products. However, they 
will still hold assumptions and goals that may interfere with their objectivity. 
Corporate representatives may habitually offer gifts or favors that should politely, 
but invariably, be declined. 

The demands of external coordination are directly related to the degree of 
external validity sought. While frustrating, time-consuming, and generally outside 
of researcher’s area of expertise, bringing external realities into the laboratory 
environment will enhance the credibility of the research and the applicability of 
the final results. 

G. ANALYTICAL STRATEGY 

Collected data should be reviewed before analysis. The variability 
introduced by involving real human participants and developmental stage 
equipment demands much time be spent in scrubbing data for accuracy and 
acceptability. The validity of a specific data point may be questionable due to 
improper data collection procedures (such as improper experimental design), 
data entry errors (such as coding errors). Fortunately, because most of the data 
collected will have been recorded to video (potentially from multiple camera 


38 



angles), it is often possible to review the original data sources and correct data 
entry errors. 

It may often be the case that groups of participants or specific equipment 
will only be available during limited time windows. Each piece of equipment to be 
tested will require participants spend training time specifically for that equipment, 
so if participants are to use more than one device, they will spend more time in 
training. Within these constraints, it may often be difficult to arrange within- 
subjects experimental designs. Worse yet, participant groups of convenience 
used for a between-subjects design, and obtained at different times, may not be 
random samples of the larger population. 

Within the many practical constraints, the most likely experimental design 
for military individual equipment usability testing is a between-subjects design, 
with randomized groups of unequal sample sizes. High between-subject 
variability should be expected. Potentially, this individual variability may be 
accounted for using pre-operation surveys of technical and military experience, 
or by testing the users in some standard, related tasks to establish baseline 
capabilities. 

The data gathered with a purely between-subjects design can be 
compared with a traditional ANOVA model (Figure 6), or a non-parametric test 
like the Kruskal-Wallis test. Non-parametric analysis provides protection against 
non-normality of the data (a strong possibility in human performance measures) 
and more statistical power in data sets with low numbers of participants. 
However, non-parametric analysis may not be possible with the more 
complicated experimental designs (such as odd combinations of within-subject 
testing) that may result from last-minute adjustments. 
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yij = A + «/ + y^y + ^/,y 
where: 

/ the device tested 

j the participant # 

Yij the task time for participanty with device/' 
a. the device effect for device / 

the participant effect for participant y 
£.j random error □ n{/j,a) 

Figure 6. ANOVA Model for Usability Measures 


The hypothesis tested is given in Table 5. The measures of performance 
used will most often be speed (task time) and / or accuracy (# of errors, or error 
rate). 


Table 5. Research Hypothesis for Equipment Testing 

Hq: Users perform the tasks equally well using the different 

equipment. 

Hi: Users perform differently using the different equipment. 


If the ANOVA or Kruskal-Wallis tests detect a difference between the 
devices, they should be followed by Tukey’s multiple comparison test or the 
appropriate non-parametric multiple comparison, to then rank the equipment. 
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IV. CASE STUDY 


A. INTRODUCTION AND RATIONALE 

The purpose of this case study is to demonstrate the applicability and 
validity of the Applied Warfighter Ergonomics Observation Laboratory for usability 
testing of military individual equipment. Four different Ruggedized Personal 
Digital Assistants (RPDAs) were obtained and their hardware and software user 
interfaces compared. According to the equipment developers, during their 
research and development phases, the RPDAs have been evaluated primarily 
using "next-bench design"; i.e., an engineer who most likely has unusually 
sophisticated technical, but no military experience, presents an interface or 
prototype to a similar engineer at the "next bench" over, who then gives the 
designer his or her feedback. 

An additional goal of this case study is to serve as a template for similar 
studies in the future. Notes on administrative and logistical minutia are included 
where they might facilitate future studies. A tremendous amount of time was 
spent during this research correcting problems that might be avoided by following 
the recommendations given here. 

The RPDAs were provided by manufacturers and the associated military 
program offices2. RPDA-D was the first device tested, because it was the first 
available in a complete and testable configuration. Initial support for the device 
was provided by Captain Francisco Caceres, USMC. Captain Caceres 
configured and provided three fully functional devices with 1:250,000 maps of the 
Monterey Peninsula. Unlike the other single-unit devices, RPDA-A consisted of a 
small, lightweight handheld unit, connected by cable to a large central unit, which 
is further connected to the communications unit with more cables. The 

2 In this report, the four devices tested are identified simply as RPDA-A, RPDA-B, RPDA- 
C, and RPDA-D. The actual device names and manufacturers are held as For Official Use Only, 
and can be obtained from the Operations Research Department at the Naval Postgraduate 
School. 
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associated hardware required by RPDA-A was carried in a backpack. Like 
RPDA-A, RPDA-B and RPDA-C consisted of a single unit. Unlike the other units 
and civilian PDAs, RPDA-C does not have a pressure-sensitive screen. RPDA-C 
uses a magnetic inductive stylus and a physically more durable screen than other 
PDAs. RPDA-B used the same hardware platform (including screen and stylus) 
as RPDA-C, but with a subset of the software build used in RPDA-A loaded onto 
it. Configured in this manner solely for experimental testing, RPDA-B had greatly 
reduced functionality compared to the other RPDAs. Therefore, RPDA-B was not 
tested using a full tactical scenario. Rather, the same group of participants 
learned RPDA-A and RPDA-B together (because those devices used similar 
software). Each participant in this group used both devices (A and B). To counter 
the order effects, a crossover design with random assignment determined 
whether participants used RPDA-A or RPDA-B first. 

B. METHOD 

1. Participants 

The intended users of the four different RPDAs analyzed include U.S. 
Army Special Forces, U.S. Army Rangers, U.S. Marines, and members of the 
U.S. Army regular forces. Participants for the RPDA studies were drawn from 
Army and Marine Corps students in "casual" status (students not currently in 
training for various reasons) at the Defense Language Institute (DLI) at the 
Presidio of Monterey (POM). These students are generally junior enlisted 
personnel in the intelligence fields awaiting formal language training. Other 
participants considered included students at the Naval Postgraduate School 
(NPS) and service members from the actual units receiving the RPDAs. Although 
a convenient pool, NPS students were rejected since, as a group, they are 
entirely officers with high military proficiency and experience, and tend to be 
more sophisticated computer users. Many attempts were made to work with 
actual members of the units intended to receive these devices, such as Army 
Rangers. Operational commitments during 2004-2005, when upwards of 176,000 
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service members were deployed in Iraq for an average of 320 days each, 
precluded personnel being available for academic research. 

Working with actual participants from a targeted population (as opposed to 
a population of convenience, such as other NPS students, or local civilians), 
introduces a number of challenges to coordination and logistics. For example, 
one-third of the DLI students initially selected, trained on a specific RPDA, and 
ready to be tested, were withdrawn from the study at the last minute due to other 
demands on their time. An entirely new group had to be coordinated and trained, 
wasting weeks of effort. Table 6 breaks down the effort spent just in collecting the 
data in this study. This estimate does not include the initial time required to learn 
the lab equipment or any of the later data analysis. The biggest time savings 
could be realized by arranging for somewhat larger groups (8-15 would be ideal) 
and by having a separate duty expert provide support for and conduct the 
training on the devices. 


Table 6. Estimated Time Spent in Data Collection 

Activities per device tested 

Est. Time 

Coordinate training and acquisition with owning agency 4 hrs 

Learn to set up, operate, and troubleshoot device 

8 hrs 

Configure and set up lab for testing 

1 hr 

Develop, plan and rehearse training for participants 

6 hrs 

Activities per testing group of 3-10 participants 

Coordination with parent unit or agency 

1 hr 

Transportation and classroom logistics 

2 hrs 

Conduct training and practical application 

3 hrs 

Distribute and collect surveys and consent forms 

0.5 hrs 

Refresher training on testing day 

0.5 hrs 

Activities conducted with each participant 

Set up lab equipment for this run 

0.25 hr 

Run scenario 

0.75 hr 

Coding each run 

1.5 hr 

Approximate totai time spent 

With 4 devices, and 4 groups of 3-6 participants 

120-150 hrs 
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There were complications in this study due to high participant attrition and 
reduced availability of those participants who were able to take part in the entire 
study. For example, the first group of participants (using RPDA-D) consisted of 
19 individuals, including Soldiers, Marines, NCOs, and Officers. After having 
invested about 30 hours in coordinating and conducting their training, the 14 
junior enlisted soldiers were withdrawn from the study for a work detail, leaving 
only an officer, an NCO, and the Marines. 

2. Apparatus and Instruments 

The laboratory equipment described in Part II was used for this case 
study. Appendix J contains a detailed description of the settings of each piece of 
equipment and software used during the case study. More general notes about 
the setup follow. 


a. Recording Video 

It is important follow the checklist in Table 4. Failure to do so 
usually resulted in some sort of problem. An “X” taped to the floor marked a 
location in the lab where the participant would stand and be in a good view from 
both cameras (front and side). The ceiling microphone was strung over that spot, 
suspended approximately seven feet from the floor. Audio equipment 
(microphone gain and loudspeaker volume) were checked every time, as these 
settings are often changed between participants or between days. 

One of the most useful views of a participant using a handheld 
device is a split-screen shot, with the participant on the left side of the screen and 
a view of the equipment they are using to their right (see Figure 7). To 
accomplish this, the Mobile Device Camera (MDC), observing the RPDA’s 
screen, was carefully adjusted to capture the RPDA screen butted up against the 
extreme left side of the camera’s view. Then, the video mixer was set (transition 
#7) to push input 3 (from the MDC) across input 1 (the front camera), just until 
the entire RPDA screen was in view. Finally, the camera controller is used to 
adjust camera #1 until the participant is appropriately positioned in the remaining 
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area. There are two important items when the video mixer is set up in this 
fashion. With the T-bar in an intermediate position: 1) the inputs cannot be 
changed (you cannot switch to camera 2, for example), and 2) you must be very 
careful not to nudge the T-bar, as this will throw off your split-screen shot. 



Figure 7. Participant and RPDA Split-Screen View 

The wall-mounted cameras were adjusted and presets for each 
camera reset for each individual participant to account for differences in height. 
This initial setup took about five minutes per participant. Each recording session 
began with the participants stating their participant ID number, to facilitate 
organizing the videos later. 

3. Procedure 

During Tactical Network Topology field experiments at Camp Roberts, 
interviews and focus groups were conducted to determine how users perceive 
the usability of military PDAs, their current and potential functionality, and what 
barriers exist to adoption. A functional task analysis of users operating RPDAs 
was conducted. The results of the functional task analysis lead to the 
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development of the scenario and the specific tasks, conditions, and standards 
with which to perform the usability testing. 

The RPDAs were compared using a between-subjects experimental 
design under laboratory conditions. RPDA-D was the first device tested. A group 
of 19 participants, including staff and students in casual status at DLI, were 
trained by Captain Koichi Takagi USMC, 1LT Derek Read, USAF, and 1LT Matt 
Simpson, USAF, in a classroom on the Presidio of Monterey. All of the classroom 
instruction was recorded and is available on DVD from the Applied Warfighter 
Ergonomics Center at the Naval Postgraduate School. First, a description of the 
research was presented (see Appendix H), and administrative details were 
addressed. Using the Noldus Mobile Device camera attached to an RPDA-D unit. 
Captain Takagi conducted a fifteen-minute class on the functionality and 
procedures for operating RPDA-D. After that, the students paired off and 
practiced the user tasks using the Monterey peninsula maps loaded onto RPDA- 
D, and the practical application test sheet given in Appendix G, section 5. While 
waiting to use the few RPDA units available, the rest of the students practiced 
their conventional skills using paper maps, protractors, and the same points on 
the map to be used in the test scenarios. 

The primary hypothesis tested using this procedure is shown in Table 7. 


Table 7. Hypothesis Hi: Differences between RPDAs 

Ho: Users perform the tasks equally well using the different 

RPDAs. 

Hi: Users perform differently using the different RPDAs. 


a. Interviews and Focus Groups 

An ad-hoc, non-attributional military PDA focus group was 
conducted 1600 - 1730 on 22 Feb 2005, at Camp Roberts, CA during a Tactical 
Network Topology field exercise. The participants were: three senior Special 
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Forces staff NCOs participating in a military RPDA evaluation exercise, one 
representative from USASOC program office (but not the Mobile Access 
Infrastructure program office responsible for military PDAs), and one employee of 
an RPDA development company. 

The goal was to determine possible requirements and capabilities 
for handheld military computing devices (military PDAs) in a “brainstorming” 
environment. Ideas were encouraged regardless of current feasibility or real 
applicability. Appendix J contains a free-form description (but not a verbatim 
transcript) of the topics discussed, including possible operational requirements 
for military PDAs, general and specific functionality that military PDAs might 
have, possible uses for and tactical employment of military PDAs, and random 
comments. Some participants had not used military or civilian PDAs to any great 
extent, some were unfamiliar with the military tasks discussed. Non-attribution 
was emphasized, and it was understood that the discussion would not generate 
obligations by the contractor or the government. 

b. Surveys 

The pilot surveys were given to other NPS students and evaluated 
by Dr. Nita Lewis Miller and Dr. Jeff Crowson during the OA3402 Research 
Methods course. Over a several month period, several revisions of the pilot 
surveys were evaluated for brevity, clarity, and relevance. As a result of the pilot 
surveys, questions were rewritten, response anchors revised, the format was 
cleaned up, and several questions were discarded. The final form of surveys 
captures the participant responses with minimal demands on the participant (for 
time and recall) and the researchers (for coding and interpretation). Due to 
discrepancies in how questions were interpreted between participants, the 
wording of several questions was slightly modified between groups. In some 
cases, this precludes direct comparison of the survey results between the 
groups. For example, question number 22 on the Technology Experience Survey 
originally read, “I send and receive email messages_times per week.” This 

question was answered consistently during the pilot surveys, where the 
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administrator was available for questions. The participants during the case study 
took the surveys home. Some answered in terms of number of email sessions, 
and some in terms of individual messages. The question, and all similar 

questions, were changed to read “I send and receive email messages_ 

hours per week.” Although this renders the actual data from those survey 
questions invalid, the survey is now more robust for use in future studies. 

Organization of paperwork required for this research was facilitated 
with a portable filing system using separate, color-coded manila folders labeled 
as shown in Table 8. 


Table 8. Surveys and Related Paperwork 

1. Prepared packets (paper-clipped packets to hand out to 
participants, containing all the necessary blank forms 2-6 
below) 

2. Participant Information Forms 

3. Maps to NPS 

4. Blank Participant ID Forms and Participant ID Cards 

5. Blank Consent Forms (3 pages) 

6. Blank Technology Experience Surveys 

7. Blank Handheld Device Evaluations 

8. Briefs and Lesson Plans 

9. Teaching Materials and Practical Application Handouts 

10. Scenario Maps and Instructions 

11. Completed Consent Forms 

12. Completed Participant ID Forms 

13. Completed Technology Experience Surveys 

14. Completed Handheld Device Evaluations 


Pre-operation Surveys. Users completed the Technology 
Experience Survey (Appendix E) before using the equipment to assess their 
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demographic background, relevant technical experience, and relevant military 
skills. Although it had been administered in several previous pilot studies, during 
this study, the survey was adjusted in several ways as described above. Some 
questions therefore cannot be directly compared between users. 

Post-operation Surveys. Users completed the Handheld Device 
Evaluation (Appendix E) after using the equipment to obtain quantitative and 
qualitative feedback from the user on the devices. The survey results were used 
in conjunction with the usability testing to determine how well users' impressions 
of their performance and satisfaction with the equipment corresponds to their 
measured performance. The primary measures of interest on this survey are 
overall satisfaction with the device, and a rating of ease-of-use. Many questions 
on this survey are intended to be broadly applicable to field use of RPDAs and 
other devices, and so elicited “Not Applicable” responses in the laboratory 
testing. 


c. Usability Observations 

Tasks. The basic tasks listed in Table 9 were chosen as being 
tactically relevant, familiar to typical users, easily taught in a brief session, 
performed by both current technology and RPDAs, and common to all of the 
devices studied. Table 10 and Table 11 list tactical tasks that were identified as 
being of possible interest to similar studies. 


Table 9. Basic Tactical Tasks Used in this Study 

1. Perform a communications check. 

2. Plot a route with two legs. 

3. Read three grid coordinates from a map. 

4. Determine the bearing and range from one point to another, 
and the total distance of a route. 

5. Send a message. 

6. Receive a message. 

7. Retrieve and read back a previous message. 
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Table 10. Tasks considered, but rejected from this study 

1. Interpret streaming video data. This task was rejected as not 
being performed by existing technology, and not immediately 
familiar to typical users 

2. Send / receive digital voice messages (Voice over IP or 
other technology). This task was rejected as not being 
implemented by all RPDAs, and as unlikely to be a major 
decision criteria for any RPDA. Users were emphatic that 
they would prefer to use existing voice communication 
radios. RPDA digital communication links may not be well- 
suited for critical voice communications for because of the 
propagation characteristics of the radio frequencies used. 

3. Send / receive sophisticated map overlays or multimedia 
files. Although of considerable tactical relevance and 
interest, this task was rejected as being overly complicated 
to teach to users. 


Table 11 . Tasks for consideration in future studies 

1. Receive a situational update, and brief it to subordinates. 

2. Create a tactical map overlay. 

3. Send a map overlay. 

4. Observe a video or situation and send an intelligence report. 

5. Make a tactical assessment based on tactical images or 
video received. 

6. Receive and process intelligence reports. 

7. Establish a (currently non-existing) communications link. 

8. Send standard tactical messages (medevac, nine-line brief, 
call-for-fire, etc.) 

9. Send administrative or logistic messages. 

The users performed these tasks both conventionally, (for example, 
reading a standard paper military 1:50,000 scale map), and using alternate 
means; in this case, using an RPDA. The research hypothesis of this within- 
subjects experimental design is shown in Table 12. During the course of the 
study, it was found that users performed the tasks in very different ways, 
requiring several individual navigation tasks to be combined into a single larger 
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measurement of mapping skill. Therefore, the conventional and alternate tasks 
no longer corresponded well with each other, and the hypothesis presented in 
Table 12 could not be directly tested. To address hypothesis H 2 , the test scenario 
must be designed in tight conjunction with the device capabilities and tuned in 
pilot studies to ensure that the tasks correspond well between the conventional 
and alternate means. The seven chosen tactical tasks were coded for data entry 
as shown in Table 13. 


Table 12. Hypothesis H 2 : Conventional vs. Alternate Means 

Hq: Users perform the tasks equally well using conventional and 

alternate means. 

H 2 : Users perform differently using conventional and alternate 

means. 


Table 13. Coded tactical tasks 


Task# 

Description 

Means 

Abbreviated 

Description 

1 

Perform a communications check 

Conv. 

Alt. 

1c comm chk 
la comm chk 

2 

Plot a route with two legs 

Conv. 

Alt. 

2c plot rte 

2a plot rte 

3 

Determine the grid coordinates of three 
locations from a map 

Conv. 

Alt. 

3c read grid 

3a read grid 

4 

Determine the bearing and range from 
one point to another, and the total 
distance of a route (when supported) 

Conv. 

Alt. 

4c brg rng 

4a brg rng 

5 

Send a message 

Conv. 

Alt. 

5c send msg 

5a send msg 

6 

Receive a message 

Conv. 

Alt. 

6c rev msg 

6a rev msg 

7 

Retrieve and read a previous message 

Conv. 

Alt. 

7c read back 

7a read back 


As the testing progressed, inconsistencies were noted that required the 


task definitions and scoring standards be re-defined. For example, in the first part 

of the scenario, participants were instructed, in three separate steps, to plot a 
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route, to read three grids, and to compute a bearing and range. Although the first 
group of participants performed the task consistently, subsequent participants 
would sometimes calculate all the required information in the first step, so that 
the task time for the first step was much longer, and the subsequent task times 
were virtually zero. The task was then re-defined and the observations were re¬ 
scored to collect all the task time of the three steps into a single task, whether 
they had broken it up or not. Although some granularity is lost in doing this, the 
task can now be reliably compared between participants. 

Another problem arose only after observing many participants. During the 
initial and refresher training and practical application, about ten points were 
identified on the paper and PDA maps including all six of the points used in the 
scenario. During the testing phase, some participants were still confused about 
some of the locations, especially the Monterey Airport. This confusion resulted in 
extremely long task times as the participants scrolled around the PDA screens in 
a fruitless search. When directed to the location, they were able to complete the 
task. These data points are outliers in the sense that the are the sum of two 
behaviors: tactical skill task time (the behavior of interest), and search time (an 
error term). Rather than discard them completely, the navigational tasks were 
redefined as "off-task" after the participant had been subjectively judged as "lost" 
for more than five seconds. Indications of lost behavior include looking directly at 
the target and then away from it, and searching in an entirely wrong direction for 
more than five seconds. Subjective measures of off-task behavior such as these 
should be avoided in future studies by fully inoculating participants against 
confusion during tasks. In this case, more explicitly training participants to the 
locations of interest may have helped. Inter-coder and intra-coder reliability 
analysis was used to ensure that the subjective correction applied (defining lost 
behavior as off-task) was applied consistently. 

Other observed behaviors: errors and posture. In addition to the 
tasks observed, the five categories of error listed in Table 14 were coded as they 
occurred. 
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Table 14. Errors by Type 


Error 

Description 

Abbreviated 

Description 

1 

Slip: unintended action, user realizes error 

slip 

2 

Mistake: intended action, user is unaware of error 

mistake 

3 

Device error: the device fails or crashes, not due 
to user action 

device err 

4 

Interface error: user has a problem with device 
input or output, such as getting device to register 
input, or reading labels or screens 

Interface err 

5 

Equipment handling error: due to using this 
device, the user spends time correcting 
equipment or has a physical difficulty handling or 
operating any piece of gear, such as dropping 
rifle, untangling cords, etc. 

eq handle err 


The tactical postures described in Table 15 were coded as they occurred. 


The specific criteria for a tactical (vice administrative) posture was: shooting hand 
on the pistol grip of the weapon and able to engage in less than one second, 
otherwise define state as admin posture. 


Table 15. Tactical Postures 


Tactical 

Posture 

Description 

Abbreviated 

Description 

1 

Tactical: The participant is immediately ready for 
a combat situation, shooting hand on the pistol 
grip of the weapon and able to engage in less 
than one second 

Tactical 

2 

Administrative: The participant is not immediately 
ready for a combat situation, for example, with his 
weapon slung over his shoulder. 

Admin 


The conventional segment of the scenario was identical for each RPDA. 


The RPDA segment was intended to be identical for each RPDA. Some RPDAs 
were not available at the time the scenario was written, and the scenario was 
adjusted slightly to accommodate different capabilities. This prevented between- 
device comparisons in those specific tasks, but the majority of tasks can still be 
compared between devices. The tasks that had to be modified can still be 
compared to the baseline performance using conventional equipment. 
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Conditions. Laboratory testing took place at the Naval 
Postgraduate School, in the Applied Warfighter Ergonomics laboratory in 
Glasgow Hall room 221. Environmental conditions and personal characteristics of 
the participants not explicitly listed, were not measured. Environmental 
conditions, such as lighting and temperature, were constant, comfortable, and 
typical of an office environment. All testing occurred between 8:00 am and 6:00 
pm on weekdays. 

Test participants wore their own camouflage utility uniforms and 
the older model combat equipments listed in Table 16. 


Table 16. Battle Gear Worn in the Tactical Scenario 


Item # 

Description 

Abbreviated 

Description 

1 

Kevlar helmet 

helmet 

2 

Kevlar flak vest 

flak 

3 

Load-bearing vest with magazine pouches and 
canteens 

LBV 

4 

Rubber M-16A2 rifle 

weapon 


Standards: Measures of Effectiveness and Measures of 
Performance. Each task has a precisely defined measure of success defined in 
Table 17. All tasks start as soon as the verbally issued instructions are 
completed, for the first time, if the instructions are repeated. If the instructions 
were completely unintelligible to the participants, then task start is when the 
instructions are issued comprehensibly. 


3 Individual equipment was temporarily loaned from the Marine Corps Detachment at the 
Defense Language Institute. Considerable time was spent unsuccessfully attempting to obtain 
current-issue equipment. 
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Table 17. Measures of Task Success 


Task 

Measure of success 

comm chk 

Participant verifies that 2-way communication is established by sending 
and receiving a simple message. 

Task ends: participant completes saying or typing “comm check” 

plot rte 

Participant performs a basic land navigation task, including plotting a 
route with two legs, recording the three grid coordinates (endpoints and 
middle waypoint), computing the bearing and range of the first leg, and 
the total distance of the route. 

Task ends: participant begins reading the completed information. 
Recorded as “off-task” while actually reading information out loud. 

Off-task when participant has been "lost" on map for five seconds, and 
scrolling around only because they cannot find a location. 

read grid 

Participant computes a six-digit grid coordinate within plus or minus 500 
meters. This large leeway is due to imprecision in identifying certain 
landmarks on the military topographical maps. 

Task ends: participant begins reading the completed information. Off-task 
when participant has been "lost" on map for five seconds, and scrolling 
around only because they cannot find a location. 

brg rng 

Participant computes a bearing within plus or minus 2 degrees and a 
range within plus or minus 500 meters. Total distance also will have an 
accuracy of plus or minus 500 meters or 5%, whichever is higher. 

Task ends: participant begins reading the completed information. Off-task 
when participant has been "lost" on map for five seconds, and scrolling 
around only because they cannot find a location. 

send msg 

Participant sends a message that would be unambiguously understood 
by another military member in the current context. Abbreviations and 
tactical terminology are acceptable. 

Task ends: participant completes reading the message out loud. 

rev msg 

Participant demonstrates that he or she understands a message by either 
performing an instruction or reading it aloud to the notional patrol 
members. Paraphrasing is acceptable if the precise meaning is retained. 

Task ends: participant indicates that they have understood the message 
by performing an appropriate action, or when they have completed 
copying down information. 

read back 

Participant locates and reads back a message previously sent or received 
that he or she would not likely be able to recall from memory. 

Task ends: participant begins reading the message out loud. 


The metrics used to evaluate behaviors were task time, 


number of errors by type, and tactical posture. Task time began when the 
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scenario narrator finished reading the task (the first time, if a long instruction was 
repeated). Task time ended (that is, the participant was coded as "off-task") 
when the participant began the next behavior, which was usually reading 
something (like a grid coordinate) aloud to his notional patrol members. There 
were occasions when task completion times is in doubt, usually because the 
participant appears to have cognitively moved off the task, but forgot to perform 
the ending behavior. For example, when reading a message on the RPDA, the 
task ending behavior is to read the message aloud to the notional patrol 
members. Occasionally, the participant would read the message but not read it 
aloud, although he or she clearly had understood the message and would then 
read it aloud when prompted. A “best-guess” in these situations would mark the 
end of task time. 

Another ambiguous behavior to code was in exercising paper 
mapping skills. When given points on a map, some participants simply wrote 
down the points and awaited further instructions (the desired behavior). The next 
tasks were to plot the points and read the grid, and then to plot the route and 
read the bearing and range of just the first leg. Some participants performed all 
the tasks, as well as computing the bearing and range of all legs (rather than just 
the first), in a single step. This made it impossible to separate out task times into 
times for performing each of the individual tasks. Therefore, the routine originally 
envisioned as three separate tasks (plot route, read grid, compute bearing and 
range), was collapsed into a single longer task (plot route), with the task time 
starting and stopping when the participant was reading the data or receiving 
instructions. 

Scenario. The scripts for the tactical scenarios presented are 
detailed in Appendix F. The scenario used a second computer running a tactical 
decision-making game presented to the test participant on a 52” plasma screen 
with stereo speakers. The software used is called Combat Decision Range 
(CDR), and developed by the Marine Corps Warfighting Lab. The CDR was run 
on a laptop computer with S-video output driving the plasma screen in the other 
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room. The S-video output was also split off to input four of the video mixer for 
monitoring from the Observer room. 

Combat Decision Ranges (CDRs) were designed to present tactical 
scenarios to squad and platoon leaders in the Marine Corps. The CDRs present 
stressful situations demanding quick thinking by the participant. Within this study, 
the CDR was used to build a combat mindset in the participants. The tasks 
triggered for the usability study were very specific, and the participant was not 
allowed freedom of decisions such as the CDR was designed for. Therefore, 
there was an artificiality to the test environment, as the scenario was interrupted 
to issue specific instructions to the participant. The participants were briefed in 
advance that the tasks were at times tactically illogical or inconsistent with other 
parts of the scenario. 

In the scenario designed for this study, seven tasks were triggered 
twice, once using conventional means and once using the RPDA (the alternate 
means). The key tasks were triggered more than once in the conventional and 
alternate means. Repeating exactly the same scenario twice would have 
introduced learning effects into the second performance of the task. Therefore, 
the situations for triggering a task, such as, “send a short message”, were similar, 
but not identical, during the conventional and alternate phases of the scenario. 
Only one complete scenario, with conventional and alternate segments, was 
administered to the participant, and all participants experienced the exact same 
sequence of events. 

Conducting the scenario and observation required the operation of 
the computer playing the scenario videos, operating the camera controller, video 
mixer, and computer recording the video, reading aloud the participant 
instructions over the microphone, and talking on a handheld radio to the 
participant when appropriate in the scenario. Additionally, the participants were 
briefed, and assisted with donning their equipment and operating the RPDA. Due 
to all of these issues, a research assistant was needed for all recording sessions. 
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C. CASE STUDY RESULTS 


1. Interviews and Focus Groups 

Interviews and focus group discussions (details in Appendix K) with the 
developers and program managers of the RPDAs revealed several assumptions 
frequently are made about the population of users: 

1. Most users are comfortable with the Microsoft Windows user interface 
model. 

2. Most users are frequent video game players (sometimes referred to as 
the “Nintendo generation”). 

3. Most users have some degree of technical knowledge about 
computers and networking. 

4. Most users have and operate cell phones or civilian PDAs on a regular 
basis. 

As shown in the next section, the survey results in this study have not 
borne out these assumptions as being entirely consistent with our test user base. 
Direct surveys of the actual user population of these RPDAs have not been 
conducted. 

2. Survey Data 

The individual and aggregate raw survey results can be found in Appendix 
J. Several of the surveys were not turned in due to participants forgetting them or 
not having time to fill out the post-operation survey. Some results from the pre¬ 
operation survey about the participants in this study include: 

36% do not have a cell phone (5 hr/wk median talk time for 
participants who have a cell phone), 

50% have never operated PDA before, 

54% have a computer at home. 

Median age was 20 yrs old, 

84% directly from boot camp, 
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Participants did not comment as extensively on the surveys as was 
hoped. 


3. Usability Observations 

Observations were coded in the Observer software. The Observer’s 
elementary statistics functionality was used to create a file containing the total 
durations of all tasks, by individual participant and coder. This data file was 
exported to Microsoft Excel, where it was formatted into comma-separated value 
(.CSV) data files for analysis in standard statistics software. S-Plus version 7.0.3 
(Insightful Corporation, 2005) was used for all statistical analysis, as described 
below. 

rpda.orig is the unedited S-Plus data file. It contained some 
questionable data points: Due to time constraints, participant 504 experienced an 
abbreviated scenario that was not similar to the conditions used in the rest of the 
testing. Some participants tested two devices, in two complete, separate 
scenarios. A large, advantageous learning effect was presumed for the second 
device tested. 

As shown in Table 18, the S-Plus data frame rpda removed the 
questionable data points from rpda.orig. The measurements for devices 
tested second were deleted. Participant 504’s data were deleted. 
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Table 18. The S-Plus data frame rpda 

> rpda=rpda.orig[(rpda.orig$order==l) & 

(rpda.orig$id != 504),] 

> attach(rpda) 

> dim(rpda) 

[1] 10 114 

> rpda[,0:12] # a sample of what the data looks like: 



id 

dev 

order 

zl 

z2 

z3 

z4 

z5 

z 6 

z7 

11 

t2 

2 

502 

1 

1 

8.0 

264 

40.0 

78.0 

15.0 

30 

14.0 

74 

NA 

5 

505 

1 

1 

14.5 

290 

31.0 

112.5 

15.0 

44 

6.0 

40 

NA 

6 

506 

1 

1 

14.0 

321 

29.0 

111. 0 

23.0 

51 

18.0 

26 

NA 

7 

501 

2 

1 

5.0 

502 

100.0 

89.0 

5.0 

52 

3.0 

NA 

NA 

9 

503 

2 

1 

16.0 

318 

23.0 

49.0 

25.5 

32 

32.5 

NA 

NA 

13 

513 

4 

1 

30.5 

307 

32.0 

80.0 

15.5 

21 

12.0 

22 

55 

14 

514 

4 

1 

11.5 

258 

12.0 

85.5 

36.5 

43 

11.0 

22 

46 

15 

515 

4 

1 

15.0 

286 

31.0 

84.0 

15.0 

18 

5.0 

2 

92 

16 

527 

4 

1 

10.0 

189 

2.5 

37.5 

63.5 

29 

3.0 

1 

46 

17 

528 

4 

1 

7.0 

231 

5.0 

38.0 

18.0 

27 

1.0 

1 

42 


Measurements were times (in seconds) to complete a specific military 
task. There were seven tasks, as described in Table 13. Conventional tasks were 
denoted tasks Zi through Z 7 . Tasks performed on the RPDAs were denoted ti 
through t?. The tasks were generalized into mapping skills tasks (t 2 - t 4 ) and 
communications tasks (ti, and ts - ty). 

A device may perform well on some tasks, but poorly on others. In some 
cases, it was possible to move functionality from one device to another, to create 
a new device that performs some tasks like RPDA-A, and other tasks like RPDA- 
D. Therefore, the best performance in each task individually was of interest, 
rather than an overall rating of the devices. Using separate evaluations for each 
task also allowed decision makers to decide for themselves which were the most 
important tasks. 

Tasks 2, 3, and 4 were very closely related mapping skills tasks. The 
conventional measurements were treated as three separate measurements of 
mapping skills, and combined into a single mapping skills score. They were all 
adjusted to the scale of conventional task 2 (the Z 2 measurements), and 
averaged as shown in Figure 8 : 
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mapskill = — 


Z2 




\ 


J 


> rpda$mapskill = (z2 + median(z2,T) / median(z3,T) * z3 

+ median(z2,T) / median(z4,T) * z4 ) /3 


Figure 8 . Mapskill Function (algebraically and S-Plus code) 


The medians, rather than the means, were used for scaling, as the 
measured task times have heavy right tails. The participants with the longest task 
times also had the greatest variability in time, and the median was chosen as a 
more stable estimate of central value. 

The mapskill score was used as a composite measure of an individual’s 
conventional mapping ability. It had the advantage of incorporating three 
separate measurements from an individual. The graph below, and the 
regressions following it, show how the mapskill score was related to the 
individual Z 2 , Z 3 , and Z 4 measurements, and how those skills were related to each 
other. 
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> pairs(rpda[,c("mapskill","z2","z3","z4")],pch=16,cex=l) 
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> summary(Im(z2~mapskill,data=rpda)) 

Multiple R-Squared: |0.88 8| 

F-statistic: p-value is 0.000 

> summary(Im(z3~mapskill,data=rpda)) 

Multiple R-Squared: |0.94 6| 

F-statistic: p-value is 0.000 

> summary(Im(z4~mapskill,data=rpda)) 

Multiple R-Squared: |0.3 98| 

F-statistic: p-value is 0.050 

Figure 9. Pairs plots of mapskill, Z 2 , Z 3 , and Z 4 


Participant skill in mapping tasks on an RPDA was expected to be partially 
related to their skill in performing similar tasks using conventional means. The 
plots below show that there may be some component of mapskill in the 
measured task times Z 2 , Z 3 , and Z 4 . These plots also show that the tasks 
themselves may be independent of each other, and also that there were some 
measurements with extremely long task times, relative to the other tasks. 
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If a statistically significant difference is found between the devices, the 
cause of the difference may be that one device is actual better than another. The 
most important reason a difference might be seen, other than due to the different 
devices, is that participants are different from each other. If the groups testing 
each device appear to be dissimilar due to small sample size or inspection of the 
participants, an adjustment must be made for participant effects. 

A naive heuristic for adjusting for this participant effect is to form a ratio of 
the RPDA task time to the mapping skill score (which are both times in seconds). 
The results of this study do not directly support this transformation, however, it is 
proposed as a way to move forward with the evaluation while adjusting in some 
way for participant effect. 

> rpda$t2adj = t2 / mapskill 

> rpda$t3adj = t3 / mapskill 

> rpda$t4adj = t4 / mapskill 
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Unlike the closely related conventional navigation tasks (Z 2 , Z 3 , and Z 4 ), 
which could be viewed as separate measurements of more general mapping 
skill, the communications tasks were less like each other. It is not clear that a 
single skill score is appropriate with them. Task times for the communications 
tasks are therefore adjusted directly from a ratio with the original task. 


> rpda$tladj = tl / zl 

> rpda$t5adj = t5 / z5 

> rpda$t 6 adj = t 6 / z 6 

> rpda$t7adj = t7 / z7 


Several statistical tests for a difference between the three devices can be 
performed. An analysis of variance (ANOVA) model allows incorporation of more 
independent variables into the model, such as test order or survey responses. 
One possible ANOVA model in a study such as this one is given in Figure 11. 


Vi.iM 

-// + «,+ p. +9^+ 

where: 

/■ 

the device tested {A, B, D} 

i 

the participant # {0500...0599} 

k 

the order tested {1 or 2} 


the task time for this combination 

a. 

the device effect for device /' 

A 

the participant effect for participant j 


the order effect for order k 


random error □ n{ju,(7) 

Figure 11. 

General ANOVA Model for Usability Testing 
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Using ANOVA, a difference was not detected based on task 3 (read grid 
coordinates from a map): 


> anova (aov (tS^^dev, data=rpda) ) 

Analysis of Variance Table 

Df Sum of Sq Mean Sq F Value Pr(F) 

dev 2 29082 14541 2.14 |0.188| 

Residuals 7 47559 6794 

As expected, when adjusting for participant effect, the statistical 
significance of the test was even lower: 


> anova (aov (tSadj'^dev, data=rpda) ) 

Analysis of Variance Table 

Df Sum of Sq Mean Sq F Value Pr(F) 


dev 2 
Residuals 7 


0.221 0.110 
0.452 0.065 


1.71 0.248 


In task 5 (the unadjusted time to send a message), a statistically 
significant difference was found (see Figure 12), and it was initially concluded 
that using the device RPDA-D is in fact faster than using RPDA-A (and B is also 
better than RPDA-A at a slightly lower confidence threshold): 
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> t5 . aov=aov (tS'^dev, data=rpda) 

> anova(t5.aov) 

Analysis of Variance Table 

Df Sum of Sq Mean Sq F Value Pr(F) 

dev 2 16999 8499 5.57 

Residuals 7 10689 1527 


0.0357 


> multicomp(t5.aov) 

95 % simultaneous confidence intervals for specified 
linear combinations^ by the Tukey method 

Estimate Std.Error Lower Bound Upper Bound 


A-B 

98. 

. 6 

35. 

. 7 

-6. 

.47 

204 . 

.0 

A-D 

85. 

. 6 

28. 

.5 

1. 

.59 

170 . 

.0 

B-D 

-13. 

.0 

32 . 

. 7 

-109. 

.00 

83. 

.3 


> plot(multicomp(t5.aov)) 
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} 
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-120 -80 -40 0 20 40 60 80 120 160 200 

simultaneous 95 % confidence limits, Tukey method 
response variable: t5 


Figure 12. ANOVA analysis of task 5 


However, when the time was adjusted by the individual’s conventional 
task time, a difference was not seen. Therefore, the better task times may not be 
due to better devices, but to better operators, better training, or some other effect 
(see Figure 13). 


> anova (aov (tSadj^^dev, data=rpda) ) 

Analysis of Variance Table 

Df Sum of Sq Mean Sq F Value 

Pr (F) 


dev 2 

91 45.3 1.81 

0.232 


Residuals 7 

175 25.0 



Figure 13. 

ANOVA analysis of task 5, adjusted for participant 
effect 


There were also slight differences in the scenario presented during the 
testing of RPDA-D and of RPDA-A. RPDA-B used a very abbreviated scenario 
with only two tasks. Although the tasks were similar, the RPDA-B scenario was 
not at all comparable to the scenario presented during the testing of RPDA-A and 
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RPDA-B. This additional confound complicated statistical analysis to the point of 
making it difficult to justify with such small sample sizes. 

ANOVA is subject to assumptions of normality and of equal variances. 
Both assumptions are highly suspect with this data. First, the data are known to 
not be normally distributed. Task times cannot fall below zero, yet some tasks 
have short task time and high variability. The tasks are also seen to have heavy 
right tails, with occasional users taking extremely long times. 

When using ANOVA on data such as these, variance-stabilizing 
transforms may be considered to address the impact of unequal variances. 
These data appeared to become more normally distributed with both logarithmic 
and even square-root algorithms, but with the very small sample sizes, the 
appropriate transform is unclear. 

To ensure against unwarranted assumptions, a two-sided Kruskal-Wallis 
rank sum test was performed on the same data. Rank-based tests are especially 
appealing because of the heavy tails of the distributions. Figure 14 demonstrates 
non-parametric testing of task 3, task 5, and task 5 adjusted for participant effect. 


> kruskal.test(t3, dev) 

Kruskal-Wallis chi-square = 

4.170, 

df = 

2, 

p-value = |0.124| 

> kruskal.test(t5, dev) 

Kruskal-Wallis chi-square = 

6.070, 

df = 

2, 

p-value = 0.048 

> kruskal.test(tSadj, dev) 

Kruskal-Wallis chi-square = 

3.386, 

df = 

2, 

p-value = 0.184 


Figure 14. Kruskal-Wallis tests for device effects 


When using S-Plus, Kruskal-Wallis multiple comparisons are performed 
manually. Since a difference was detected, the multiple comparisons were 
performed on the task 5 times, at a 95% confidence level (Figure 15). 
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> t5rank=rank(t5); tSrank 


# compute the ranks 


[1] 10.0 9.0 8.0 5.0 1.0 6.0 2.0 3.5 7.0 3.5 

> dev 

[1] AAABBDDDDD 


# initialize array 


> R=c (0,0,0) 


> R[l]= sxim(t5rank[dev=="A"] ) ;R[1] # compute the rank sums 

[1] 27 

> R[2]= sum(t5rank[dev=="B"]);R[2] 

[ 1 ] 6 

> R[3]= sxim(t5rank[dev=="D"] ) ;R[3] 

[ 1 ] 22 

> n=c(length(tSrank[dev=="A"]), length(tSrank[dev=="B"]), 

length(tSrank[dev=="D"])); n # n[i] = number of obs. 

[1] 3 2 5 

> N=sxim(n) ; N # N = total obs. 


[ 1 ] 10 

> k= length(levels(dev)) 

> alpha = 0.05 


# number of treatments 
# desired confidence level 


> S. squared =1/(N-1) * (sxim(t5rank^2) - N*(N+1)^2 / 4); 

S. squared 

[1] 9.111 

> T.stat = (sum(R^2/n) - N*(N+1)^2 / 4) / S.squared; T.stat 

[1] 6.069 

> student.t.stat = qt(l-alpha/2, N-k); student.t.stat 

[1] 2.36 

> constant=student.t.stat * sqrt(S.squared * (N-1- 

T. stat)/(N-k)); constant 

[1] 4.62 

The following lines display the left-hand side and RHS of the 
multiple comparison criteria for the Kruskal-Wallis test. If the LHS > 
RHS, then there is a difference. 

> i=l; j=2; c( abs(R[i] / n[i] -R[j] / n[j]) , constant * 

sqrt (l/n [i] + l/n[j]) ) 

[1] |6.00| 4.22 

> i=l; j=3; c( abs(R[i] / n[i] - R[j] / n[j]) , constant * 

sqrt(l/n[i] + l/n[j]) ) 

[1] |4 ■ 60| 3.37 

> i=2; j=3; o( abs(R[i] / n[i] - R[j] / n[j]) , constant * 

sqrt(l/n[i] + l/n[j]) ) 

[1] 1.40 3.86 

Figure 15. S-Plus Code for Kruskal-Wallis Multiple Comparisons 

Tests 
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At a = 0.05, B is better than A, and D is better than A. However, B is no 
better or worse than D. As with the ANOVA test, however, if the heuristic 
adjustment for participant effect on task 5 is performed, a statistically significant 
difference is not seen: 


> kruskal.test(tSadj, dev) 

Kruskal-Wallis chi-square = 3.68, df = 2, p-value 


0.159 


4. Inter-Rater Reliability 

To verify that the measurements are objectively coded, inter-rater 
reliability (IRR) analysis was performed. Task time assessments by two different 
researchers (“A” and “B”) on the same video file (participant 0513 operating 
device RPDA-D) are shown in the "confusion matrix" in Table 19. 


Table 19. 

Confusion Matrix for Participant 

0513, RPDA-D 


t1 

t2 

t3 

t4 

t5 

t6 

t7 

Error 

Total A 

ti 

22.63 

- 

- 

- 

- 

- 

- 

0.33 

22.96 

t2 

- 

54.73 

- 

- 

- 

- 

- 

0 

54.73 

t3 

- 

- 

9.71 

- 

- 

- 

- 

0 

9.71 

t4 

- 

- 

- 

25.1 

- 

- 

- 

0.14 

25.24 

t5 

- 

- 

- 

- 

92.06 

- 

- 

0 

92.06 

t6 

- 

- 

- 

- 

- 

19.53 

- 

5.8 

25.33 

t7 

- 

- 

- 

- 

- 

- 

5.5 

0.94 

6.44 

Error 

0 

1.63 

1.65 

0 

0.17 

0 

0 

- 

3.45 

Tot.B 

22.63 

56.36 

11.36 

25.1 

92.23 

19.53 

5.5 

7.21 

239.92 


Table 19 shows the total duration (in seconds) scored for the seven tasks 
(t1 - t7) by two different coders (“A” and “B”). The actual time coded by coder A 
is shown in the Total A column at the right, and the actual time coded by coder B 
is shown in the Total B row at the bottom. The number in the matrix reflects the 
agreement between the two times, that is, the lesser of the two times. For 
example, both coders agreed that the participant spent at least 22.63 seconds on 
task 1. Coder A actually measured the time as 22.96 seconds. The 0.33 seconds 
of difference is shown in the Error column for A. Other types of inter-rater 
reliability analysis could have additional times in the off-diagonal cells of the 
confusion matrix, showing what task or behavior the coder recorded during the 
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times of disagreement. The frequency and sequence of observed behaviors can 
also be compared. 

Figure 16 shows that the task times (t1-t7) assigned by different 
researchers (on the vertical and horizontal axes) are in very good agreement for 
this observation. 



Figure 16. Tasks times as coded by Rater A and Rater B 


Statistics commonly used to ensure that objective criteria are being 
applied in scoring the behaviors include the index of concordance and Cohen’s k. 
The index of concordance is the proportion of agreements to total observed time. 
Cohen's k is a measure of the proportion of agreement that exceeds that 
expected by chance (Figure 17). 


^-Pc 

where: po = the observed proportion of agreements 

Pc = the proportion of agreements expected by chance 

_ Figure 17. Formula for Cohen’s k _ 


In this comparison of raters A and B, the total duration of agreements is 
229.3 seconds, the total duration of disagreements is 10.66 sec, the index of 
concordance is 0.96, and Cohen’s k = 0.94. This indicates that independent 
observers using the same instructions were quantitatively coding behavioral data 

in a very reliable manner. Achieving this level of agreement requires detailed and 

70 







objective coding criteria and scenarios that provide for precisely defined task 
start and end points. 


D. DISCUSSION OF CASE STUDY 

1. Conclusions from Case Study 

Due to the high rate of participant attrition, statistically supportable 
conclusions cannot be drawn from the quantitative data collected in this study. A 
number of usability issues were directly observed or were noted in the post¬ 
operation surveys. 

a. All RPDAs interfered with the user’s ability to use their personal 
weapon. (HDE 0501AB, 0502A, 0505AB, 0506A, 0515D, 
0528D, RA0502 at 20:14; RA0503 at 28:15)4 

b. Participants were generally not particularly satisfied or 
dissatisfied with a stylus input device, although some were 
highly opposed. Almost all participants who had negative 
comments about the stylus had never used one before. (HDE 
0501AB, 0502B, 0505AB, 0506A and corresponding TESs) 

c. User satisfaction was not tied to objective performance. For 
example, participant 0502 was "Very Dissatisfied" with the 
RPDA-B stylus, and "Very Satisfied" with RPDA-A. However, he 
was actually faster in the messaging tasks with RPDA-B than 
with RPDA-A. The video shows him repeatedly tapping with the 
stylus using RPDA-B, which is a source of frustration but was 
not detrimental to his overall performance. 

d. A single incident can disproportionately affect user satisfaction 
with the overall device. Participant 0501 experienced a single 
incidence of difficulty with the stylus, but no other problems. His 
satisfaction with the stylus was "Very Dissatisfied", although his 
performance using it was average. 

e. RPDAs can bring inexperienced personnel, who performed 
poorly on baseline tasks, up to performance levels similar to 


4 The parenthetical comments indicate the survey questions or the observations that support 
the conclusion. Surveys are prefaced "HDE" for the Handheld Device Evaluation or "TES" for the 
Technology Experience Survey, followed by the four-digit participant number. RPDA observations 
begin with "R", followed by the letter identifying the RPDA, followed by the four-digit participant 
number, followed by the approximate time (minutes:seconds) in the video that the behavior was 
observed. 
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personnel who excel at baseline tasks. Participants with little 
military experience ("novices") required considerable training to 
perform the conventional mapping tasks and were generally two 
to ten times slower than users with high military experience 
("experts"). The novices were also more likely to make errors in 
conventional tasks than the experts. However, novice 
performance using the RPDAs for mapping was comparable to 
expert performance. Although experts generally fared better at 
RPDA mapping than novices, they were only incrementally 
better. The novices cut their mapping skill task time by a factor 
of ten in some cases, while some experts actually experienced 
an increase in their task time compared to conventional means. 
(See task 3, participants 0502, 0503, and 0527). 

f. Users commented about the bulk of RPDA-B, which was the 
largest one-piece unit, but expressed even stronger 
dissatisfaction with the RPDA-A, which had a smaller handheld 
piece tethered to equipment in the pack (HDE 0501A). Several 
problems (one fairly serious) were observed with the tethered 
cord become tangled in other equipment (RA0503 at 28:15). 

g. Lacking feedback and appropriate error messages, users 
wasted much time attempting incorrect procedures. Virtually all 
participants indicated that if they experienced a problem using a 
computer, they would "keep trying" as their first step to solve it. 
This behavior was frequently observed (RA0501A at 37:50) 
when the device did not give feedback to the user that an 
inappropriate action (such as scrolling off of the edge of a map) 
was being attempted. 

h. The RPDAs made novice users feel more confident in their 
mapping skill, even when their performance and accuracy were 
poor. (HDE 0502A, 0505A). This may be an example of the 
false pedigree of credibility that high-technology devices can 
carry, as discussed in the literature review (page 11). 

i. The low visibility and small size of the PDA screen (72 dots per 
inch, low-contrast screen, compared to a 1200 dots per inch, 
high-contrast paper map) made it very difficult for users to 
search for locations with which they were unfamiliar. 

2. Recommendations from Case Study 

Participants in this study had only an introductory level of proficiency and 
experience with the devices. Performance at this stage should not be of primary 
interest, except to the extent that these initial impressions make an impact on 
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user acceptance. Users proficient with the stylus and the software should be 
participants in testing usability and user satisfaction. More proficient users also 
have reduced intra-task variability and would provide better data for statistical 
analysis. 

The recommendations for context-aware designs found in the introduction 
to this thesis (page17) have been confirmed by the qualitative results of this 
study: 

Sensible default behavior. Users benefited from RPDAs that were 
organized consistently and performed appropriate actions the first time. An 
example of good default behavior: tapping a location on a map brings up the 
most likely information desired (grid location). An example of poor default 
behavior: when reading a message, RPDA-D required users to scroll to the 
bottom of the message (past header information that was never used) to get the 
information they want - the message text. The header information should be less 
prominent by default. 

Restricted options. Errors were frequently seen when users are allowed 
to perform incorrect tasks, such as closing a necessary application, or changing 
vital communications settings. Users should be protected from actions that would 
be incorrect in most situations. 

Robustness to inputs. Users became extremely frustrated when simple 
slips caused serious errors. Slips appeared to occur due to users having to split 
their attention or due to input errors with the stylus. 

informative error conditions and graceful recovery. RPDAs should 
provide immediate feedback to the user when inappropriate tasks are attempted, 
or they will likely continue to attempt that task incorrectly before trying other 
solutions. This behavior was reported on almost all the surveys and directly 
observed in many instances. 

The appropriate level of detail presented on a map appears to be 
extremely important. Displaying multiple scales of map (for example, 1:250,000 
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for zoomed out and 1:50,000 for zoomed in) may help alleviate some of the 
confusion users experienced when zoomed out on a cluttered, highly detailed 
map. 

Menus and action buttons should be consistent and readily visible. Users 
were frequently observed looking in the wrong menus for functions that appeared 
in that location in another application. Users do not generally want (or need) to 
understand that different applications may be running simultaneously (such as 
separate communications and navigation applications). Different applications 
should still maintain similar interfaces. For example, when changing between 
communication and navigation functions on RPDA-A, the menu selections 
change subtly. In the communications mode, users consistently went to the 
incorrect menu (which would have been the correct menu in the navigation 
application) to change applications. 

3. Follow-on Research to the Case Study 

The framework of the study remains in place, including points of contact, 
method, and equipment. Adding additional data points with any of the devices, in 
order to allow more robust statistical analysis, should be a relatively 
straightforward process. Continued work with the RPDA case study may yield 
definitive results, allowing comparison between RPDAs, and additional 
recommendations for the field of RPDA design. Integrating the Applied 
Warfighter Ergonomics Center’s laboratory and field capabilities early on in the 
design and prototyping of RPDAs and other military individual equipment will 
result in better equipment being fielded. The AWE laboratory could be used in 
the design of RPDAs in the following ways: 

• defining the appropriate tasks to be performed by an RPDA, 

• quantifying the effect of RPDA use on battlefield situational 
awareness (SA), 

• defining the appropriate characteristics of an RPDA: device 
size, weight, screen size, reflectivity, brightness, resolution, 
and types of controls. 
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A number of questions surfaced during the planning, execution, and 
analysis of this case study that are of keen interest but beyond the scope of this 
study. The following topics would be excellent avenues to pursue a more detailed 
study using the capabilities of the Applied Warfighter Ergonomics Observation 
Laboratory. 

Technology experience among various U.S. military populations. The 

Technology Experience Survey is ready to be issued in a full study, and concise 
enough to reasonably administer to large populations without excessive 
coordination with the participating units. Some military surveys are so extensive 
as to be very demanding on operating forces with precious little time to spend 
supporting such external demands. Although web-based surveys are currently 
popular due to the ease with which they can be administered and scored, this 
survey is a particularly poor choice to be administered via electronic means. A 
large canvassing of different populations could be used in several ways. First, 
natural groupings of responses could be used to classify persons of high, 
moderate, and low technology experience, and such ratings could be used as 
independent variables in future studies. The population sizes in the case study 
were too low to adequately break down groups in this manner. Then, the actual 
technology experience of groups of interest to military technology developers 
could be quantified. Rather than assuming the likely users are of the “Nintendo 
generation,” a device targeted at Army Ranger team leaders could be designed 
from the start with actual data about the technology experience of those users. 

Baseline performance metrics of standard tactical tasks. The tactical 
tasks used in this study, or other relevant tasks, could be measured in larger 
populations to establish baseline performance metrics such as task time and 
error rate. 

Performance differences between populations of interest. In this 
study, students at the Defense Language Institute were used as surrogates for 
the real users of interest. Naval Postgraduate School students were not used 

under the untested assumption that their technology experience greatly 
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surpasses that of common military users. This laboratory could be used to 
definitively establish the performance differences between populations such as 
typical Marine infantrymen, DLI students, and civilian software programmers. 
This information could be used by developers, testers, and program managers to 
establish (or question) the external validity of studies that use test personnel from 
outside the actual populations receiving the equipment. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

With the Applied Warfighter Ergonomics Center, the Department of 
Defense now has an independent, organic capability to perform individual 
equipment usability testing, rather than relying on defense contractors and ad 
hoc testing. Military individual equipment can be scientifically evaluated under 
controlled conditions in the laboratory and realistic conditions in the field 

Innovations, such as RPDAs, that demonstrably improve productivity, are 
useless if they are not adopted by warfighters in the field. More extensive 
research may find tradeoffs or reinforcement between the factors that motivate 
adoption and the factors that drive usability. Increasing user satisfaction, even in 
ways that limit the functionality of the equipment, may result in more widespread 
adoption and ultimately a better fighting force. Feature-laden devices that 
frustrate or confuse users, and are not adopted, will be just dead weight in the 
bottom of a pack. 

This research introduces a validated, generalizable way of measuring 
what equipment works best for specific populations of users, and what equipment 
those users are most satisfied with. Both equipment and training can be 
designed so that functionality is accessible to the actual users (not just to the 
engineer on the next bench), and so that users will be inclined to fully utilize the 
equipment of their own volition. 

B. RECOMMENDATIONS 

The Applied Warfighter Ergonomics Observation Laboratory is primarily 
equipped for collecting measurements and establishing quantitative comparisons 
on individual military equipment. Nothing in the laboratory inherently lends itself 
directly to creating better designs, although it can (and should) be integrated into 
the early design, prototyping, and subsequent testing of many different types of 
equipment. Early user assessment, continuous testing and feedback, and in- 
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progress validation are key capabilities that this laboratory enables for military 
individual equipment. Appropriate program offices should be made aware of the 
laboratory’s capabilities so that ultimately, individual soldiers, sailors, and 
Marines can benefit at a personal level from improved equipment made possible 
by the AWE Center. 

The equipment in the laboratory is complicated and experienced users are 
required for the laboratory’s continued operation and maintenance. A concerted 
effort will be required to maintain continuity in student skills with the laboratory 
equipment. Incorporating various parts of the lab’s capabilities into existing and 
future courses will stimulate interest and increase the pool of capable users. 

As shown by Carpenter and Nakamoto (1994), the behavior of novice 
users and early adopters may not be indicative of mainstream behavior. Users 
will tend to adapt their preferences towards the technology as it becomes 
adopted. So, the preferences of the first users may not be a good gauge of either 
qualitative satisfaction or quantitative performance. Task times and error rates 
observed early in Jordan's (1998) learning curve are of less interest than 
measurements taken of more experienced users. More experienced users will be 
more representative of how the equipment will actually be used in combat. A 
concerted effort, involving command interest at the highest possible levels, must 
be made to commit appropriately representative users to usability testing in this 
laboratory. 

C. FOLLOW-ON RESEARCH 

The field capabilities of the laboratory have been tested in small studies at 
Camp Roberts and in mobile data collection on the NFS campus. A full exercise 
of the behavioral observation capabilities in an actual deployed environment has 
not been performed. A full demonstration of this deployable capability, to validate 
the results of a laboratory study, would complete the establishment of this 
laboratory as a premiere center for research in military individual equipment. 
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The AWE lab is also potentially a source of justifiable human performance 
data for computer models and combat simulations, providing not just point 
estimates, but empirically derived distributions. The lab can be used to accurately 
measure task times, error rates, and training effectiveness in a wide variety of 
military applications. 
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APPENDIX A. GLOSSARY AND ACRONYMS 


Some terms have a unique meaning within a specific context, such as within the 
Noldus Observer software. In those cases, the context is indicated in [brackets]. 


1:50,000 

The scale of standard military maps used 
by ground forces. Such a map might be 
called a “one to fifty”. 

AWE 

Applied Warfighter Ergonomics 

BDUs 

Battle dress uniforms worn by United 
States Army personnel. 

Bearing 

Direction from one point on a map to 
another point. Bearing may be given in 
relation to magnetic north (as read on a 
compass) or true north (as read from a 
map). 


Events that are recorded during a scenario 
are called behaviors. There are event 
behaviors, which occur at a single instant 
in time, and state behaviors, which have a 
duration. Behaviors are grouped into 
Behavioral Classes. 

Cammies Camouflage utility uniforms worn by 

United States Marines. 

CDR Combat Decision Range 

Communications Check (Comm An initial exchange used to verify that 

Check) communications gear is functioning. Users 

on each end will send and respond with a 
message. 


Behaviors [Noldus Observer 
Software] 
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Configuration [Noldus Observer A configuration should not be changed 
Software] after starting to collect data. A 

configuration is locked after data files are 
made. If you unlock the configuration, you 
must remove the data files, make changes 
to the configuration, and then add the data 
files back. 

DV Dependent variable. Measurable data that 

can be predicted by the independent 
variables (IVs). 


Grid, Grid Coordinate The standard ground combat military 

method to identify points on a map (as 
opposed to latitude and longitude, used in 
aircraft and ship navigation). A four-digit 
grid coordinate (or just “grid”) is accurate 
to within 1,000 meters. A six-digit grid is 
accurate to within 100 meters. An eight¬ 
digit grid is accurate to within 10 meters, 
and is the practical limit of accuracy in 
most current navigation devices, such as 
reading a 1:50,000 scale military 
topographical map. 

HCI Human-Computer Interaction 

HF Human Factors 

HSI Human Systems Integration 

IV Independent Variables (IVs) represent 

specific information about the participants 
and the scenario in any particular 
observation session.. 


IRB Institutional Review Board 

ISO International Organization for Standards 

(acronym in French) 
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Leg 


A single path from one point on a map to 
another point, often part of a route 


MEF 

Military individual equipment 

ORD 

PDA 

Patrol Route 


Marine Expeditionary Force 

Equipment that is normally carried and 
employed by a single person in combat 

Operational Requirements Document 

Personal Digital Assistant (handheld 
computer) 

see Route 


Projects [Noldus Observer 
Software] 


Range 


Route 


RPDA 


Scoring or Coding 


A project file contains the configuration, all 
observation and data files, and analysis for 
a given experiment. You can only do 
analysis on data within the same project. 

Straight-line distance from one point on a 
map to another point. 

A series of connected straight lines (legs) 
used to navigate from one point to a final 
destination, usually through several 
intermediate points. 

Ruggedized Personal Digital Assistant. A 
handheld computer that is designed for 
outdoor use in a variety of environments. 
RPDAs are commonly based on 
conventional Personal Digital Assistants 
(PDAs) running Palm OS, Microsoft 
Windows Mobile Edition (also called 
Windows CE or PocketPC), standard 
Microsoft Windows, or Linux. 

The process of turning observed behaviors 
into a data file for later analysis. 
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Subjects [Noldus Observer 
Software] 


TNT 


Ul 

VOIP 


Workspace [Noldus Observer 
Software] 


The roles (“billets” or "actors") played by 
the people simultaneously participating in 
an observation session, such as “Team 
Leader” and “Rifleman”. Very often there 
will be only a single subject (the person 
using the equipment being tested), and so 
subjects will not be set up in Observer. 

Tactical Network Topology. An ongoing 
NPS research project, formerly known as 
"STAN" 

User Interface 

Voice over Internet Protocol. A means to 
send digitized audio over networks using 
Internet Protocol (IP). VOIP has the 
advantage of using common networks, but 
is inefficient compared to other ways of 
sending digital audio. 

An Observer workspace is a way to 
organize what projects are visible. Use 
different workspaces for different 
researchers, or completely different 
research. The Observer system in the 
AWE center has a Sample Workspace, a 
workspace for Class Projects, and 
separate workspaces for theses and other 
research. 
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Introduction 

The Naval Postgraduate School (NPS) 

Applied Warfighter Ergonomics (AWE) 

Center’s Observation Laboratory provides a 
laboratory and established methodologies 
for scientifically evaluating the human 
factors considerations of oombat equipment 
used by military personnel under both 
controlled and realistic field conditions. 

This User's Guide provides detailed, step- 
by-step instructions and a tutorial for using the laboratory in a basic configuration 
suitable for learning the system and for conducting a small study. 


Copobilitjes 

Technical capabilities of the Observation Laboratory 
include: 

Capturing behavior in the laboratory using digital 
remote video cameras and fixed and wireless 
microphones. 

Recording professional quality digital video In a laboratory or field environment to 
computer files and DVDs, using portable miniDV camcorders, or mobile device 
camera with portable miniDV VCR, 

Mixing and recording audio and video from 
up to four sources (such as video cameras or 
video tape) onto AVI / MPEG-2 computer 
files, DVD, or videotape. 

Producing highlight videos of events of 
interest. 

Editing, producing, and analyzing 
observations using Noldus Observer 
software to score observation sessions and perform statistical analysis. 

Direct screen-capture of the participant’s computer screen to the observer's 
screen and to video. 

Coding behavior and other events to time-stamped logs that are tied to the video 
files. Directly perform statistical analysis on the logs and jump to coded events on 
video. 
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Research capabilities include: 

Observe participant(s) while using a computer or other equipment in the lab. 
Record comments during an after-action report (AAR) de-brief. 

Record interviews and focus group studies. 

Record participants reactions to projected video, or to scenarios read over 
loudspeaker. 

Perform reliability analysis between coders. 

Perform lag-sequential analysis on behaviors and events. 


Examples of possible results 

"When given the task, 'locate yourself on the [computer] map', 7 out of 16 
participants (44%) performed it correctly within the allotted 60 seconds (mean 
time to complete = 45.7s, a= 9.8s), 5 participants (31%) completed the task after 
making one or more errors (p = 88.3 s, a= 15.2 s), and 4 participants (25%) did 
not complete the task within 5 minutes (gave up or experienced a technical 
malfunction).” 


"Buttons 1 and 2 were pressed correctly on the first try 98% of the time, button 3 
was pressed correctly on the first try 87% of the time, but button 4 was pressed 
correctly on the first try only 45% of the time. Errors on button 4 included missing 
the button (28% of the time), pressing the wrong button (18% of the time), and 
dropping the device (9% of the time)." 


"During the After Action Report (AAR) de-brief, team member #3 appeared to 
scowl or otherwise disagree with the team leader's comments on 6 occasions, 
but made no verbal comment. Members 1,2, and 6 consistently reinforced each 
other's comments verbally and non-verbally." 


"The equipment load as worn interfered with the participants' ability to shoulder 
their weapons. It increased the time to move from a standing position to a 
covered, kneeling firing position by an average of 3.2 seconds (n= 21 subjects, 
average of 5 trials per subject p=0.Q43). Here are some example video clips of 
the movement with and without the carrying load, showing how it interfered." 
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How to use this manual 

This manual is designed to allow a beginning student researcher to perform a 
basic usability analysis in the AWE Usability Lab. The equipment and procedures 
can be technically challenging at first, although you should be able to fully 
complete the tutorial (including recording and analyzing a test case) within three 
hours. It is imperative that you run through at least one test recording before 
trying to conduct even a small research study. 


1. Before designing your experiment, read The basics of observational research. 

2. When ready to conduct your usability testing, follow the step-by-step tutorials 
in Preparation before an observation session and Recording a session 
(Laboratory). In these steps, you will create video files of your participants. You 
should also refer to Appendix C: Observation Lab Technical Reference. 
Appendix C details how to avoid or recover from the common problems you will 
encounter. 

3. To analyze your video files, follow the step-by-step tutorials in Scoring a 
Session and Analyzing Data. 


The other sections of this manual will be useful in configuring and upgrading the 
lab, and in guiding you in pursuit of more sophisticated use of the facility 


The basics of observational research 

To use the AWE Center Observation Lab effectively, researchers should prepare 
the following: 

1. A well-formed research question to investigate. 

2. A list of the independent variables (I Vs) that will be controlled in each 
observation session. This may include information about the participant (name, 
rank, gender, etc.) and information about the scenario or environment ("wearing 
equipment load A", "night conditions", etc.) 

3. A list of tasks to be given to the participant (e.g. "send a message") 

4. A detailed list of "behaviors" (or "events") that will be observed and recorded. 
Virtually all possible behaviors of interest should be defined before hand ("start 
task", "press button 1" "user commits error"..., "complete task") Not all events will 
necessarily occur, and some may occur multiple times. 

5. A detailed, scripted scenario, including: 

• A timeline. 

• A description of what each researcher and participant will be doing at each 
point in the scenario. 
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♦ Indie&tidns «r will tridgef evant «r bahat^ar af intefast (a g. how 
doa^lh^ participant Itnow to slar^a ta$!<?). 

* What equipment will be u&ed (i.e. helmet, fine, handheJd devioe, etc.). 

6 . Any required papervyerK sueh as iRBs. conseoi forms. becK^round 
questionnaires, etc. 


Reoommerited introductions to usability lesling include; 


Rutrin, J. (t 994 ). H^rpc/lwoft 0/ usdMriy lasting: Wov/ fo pf^n, 
tiesigf7. and conduct sftaa^ivB tes^s. New Yertc VWley. 

Jordan. P.\/y. ( 1 ^^). An introduction to usability. Sristol, PA: 

Tsylof a Francis. I no. 

Perlman, G. (2Q<55). Hd bibtfogmphy: HumBn-comp^tBrinteraction 
/ iJSiSf mtBrfsc^ ussbiiity. Retrieved August 2DD5. 2005 from 
http://www.hcibib.Drg/ 


Researchers will produce video Hies of the o bservatiqn session far each 
participant, and then use Noldus Observer to score the behaviors observed. The 
daia cen expcded as E^ccel filas. and however you please. 

Different researchers can score the same sessions from video files. ar>d 
a-ulcrnalically da reliabrirty analysis behvean the ooders. 

The main phases of ustng (he otsservauon laboratory are to: 

1. Setup IheeKpenmint 

2. Rocord the e^esion lo video 

3. Score ^he beheviors observed during the sessk»n 

4. Analyze the data. 

Practice runs before actually attemptir^ lo collect data should be conskiefed 
absolutely necessary. 



AWE DtHenffiataon Lster^ry Bdiic Us«r Guld« 
B4fare an ObfervatuMi SeuifrA 


p^4 


92 





Preparation before an observation session 

For troubleshooting mrormation, pictures, diagfams, and Turttior details about 
each piece of equipmorti see C: Obsen^ef/o/? Lab T&chmcai 

Referofloa. 


1. Genera! 

1J. Reserve ttie AWE Otoervation Lab well in advance. 

1 2. Confirm the date and lime with all participants and others irtvolved. 

1.3. Open HS3L main raom (GL-221, pJwne 656-3138), participant fGL-221 a), 

andol)€ervoF tGL-21=&. phone §56-3134) rooms 

1.3.1. Check lighting and inspect area for cleanliness and safety issues 

1.3.2. Secure any equipntenl papers, etc. that could interfere with y^ur 

or oonfiise participant. 

1.3.3. Put up ‘'RECORDING IN PROGRESS" signs and inform others in 
Ihe area or your work. 

1.4. f^evlew the script or run prollie^s) for the olsservacion(s) to be made, 

5. BrM participant's) on cameras, mjcrcphanaa and gan#rai iab prooadunas. 
H .§. Ensune consent forms are properly filled out snd on file. 

2. Set up parilcipant room |GL-221e) 

2. T. Arrange Surhllure In accordance wrth script. Pay attention to camiera 
locatians a nd try to keep the partiei panls' noma I movemeits from 
blooking the cameras. Tape marks on the noorlo indicate where 
partfoipantg ^uld ^and. 

2.2. Have en aeeietaot st-an^l on itve me rked spoi while you preset Uie camera 
pcsttioivs. 

2.3. Provide p^ per, pencils, and ot:her props {as required] 

2.4. If used, lum on compuler {powder strip, CPU, display) 

2 J Participant workstalion operations check (as applicable) 

2.5.1, Log on as local user. 

2. S.2. Check hardware and software ccnnsurat6on(sj a nd network 
cpnr^ectiviiy. 

2.5.3. Log off, or otherwise preria re wo rfcstati on fo r participant use. 

3. Set up observer room {GL'2161 
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3.1. Have script, notepad, and pens available for taking notes during the 
session. 

3.2. Turn on power: three surge protectors, power button on computer. 

3.3. Visually check power lights: surge protectors, computer, computer 
speakers, printer, event logging keyboard, two amps, VCR, two monitors, 
camera controller, and video mixer. 

3.4. Log on to the Observer workstation as Noldus Observer (no password 
required). 
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Recording a Session (Laboratory) 


The instructions starting on the next page enable you to record the screen of the 
computer workstation in the Participant room, the live video feeds from the 
cameras in that room, and the audio from the fixed ceiling microphone and the 
wireless microphone. 

If you are not using the participant computer workstation (for 
example, when doing a study on handheld devices), then: 

* Skip step 1 below 

* In step 3, set the Observer Camera Viewer window to 
size instead of size (see Appendix C: Observation Lab 
Technical Reference for instructions) 

* In step 4.2.4, select only a region around the Camera Viewer 
window using Camtasia Recorder. 

For troubleshooting tips about each piece of equipment and software, see 
Appendix C: Observation Lab Technical Reference. 
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1, up view or [lailici^rvt wAfkstation screen (i^nly ifu^F^g :h^ pA^dp^nt 

ca/npirCsf iv£3J^sfJttonj. 


1 r Launch application: Kud only VNCto ^rtkipimt 

1.2. Veri^ that you see the screen of Ihe participafit oompuler in the VNG 
window. The VNC window should be positioned rnthe upper right hand 
corner of the screen. 

2. Set up video cameras in Particfcpar^t room, 

2.1. Thts is one of file more complicated steps. See Appendix C: Obsafvation 
LeJb Tei^fiica! Refe/ojiceon the camera controller, and the video miKer 
for Iroubleshootinig and details a-n oltier video setups. 

22. Use lha earners conifoiler to lin^ up tJie mo ca nneras at tho padiciparit 
stalon as desired. You should l>e able to see their output on the 
PREVIEW vidoe rnonitof. 

2.3. Use video mixer bo set up your video feeds. 

2.3.1. Raise the T^Bar to the lop pesHion and center the joystick. 

2.3.2= Tap the "OUT SOURCE" bunon (A - front camera, B - side 
camera) to select /our camera. 

2.4. Verify that your destred video effects are displayed on the PROGRAM 
video monitor, and 'm the CameraViewer window in the nex\ step. 

3h Displa/ vidw feed from participant room on Observer computer screen* 

3.1. Launch The Obierver Camtrivl«w«r (Mcalei 

3.2. Chech commind>Llvt to start Itie video feed. 

3.3. Position Ihe CameraViewer window In the corner of Ihe VNC window 
<"piclure-in-piciure" style), or to the left of ^he VNC window (if il is 
Important not to block any of the useYs screen). 

4. Set up vkieo f screen reoordJing software 

4.1, Launch Caimm^la Rjecqrder 

4.2. initial Setup of Camtasla Recorder - you should always record a brief fS- 
10 5eco nd) les( video to verity your a udio (Murd check) and video 
configure!lorn, (shipto step 4.3 for subsequent observations) 

4.2.1. Position the Canniasia Recorder wir^ow below and to the la ft of the 
other windows (completely oul of the way) 

4.2.2. {0 ptional) Veniy confrgurahon in Tools > options (see Ihe Appemdtx 
C: Observaf/on Leh TechnicAt Retereme ssection) 
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4.2.3. Capture > Wizard, choose Region of the Screen, check Record Audio, 
click Next. 

4.2.4. Click the Seiect Region button, and click and drag from the upper 
right corner of the screen down and left, so that both the VNC window 
and the CameraViewer window are in the capture region. Click Next. 

4.2.5. Audio device should be set to SoundMAX Digitai Audio, (or other built- 
in sound card), and Audio input Source to Line in. input Levei should be 
at maximum. Click Next. 

4.2.6. Check the Disable display acceleration box, check After save, play the 
video, and t/ncheck After save, edit the video. Click Finish. 

4.3. Record the session using Camtasia Recorder 

4.3.1. Click the Record icon ^ (or press F9) to start recording. 

4.3.2. Click the Pause icon (or press F9) to pause the 

recording. 

4.3.3. Click the Stop icon ^ (or press F10) to complete the recording. 

Be careful: clicking the Delete icon ^ will completely delete your 
recording session! 

4.3.4. Save the video file (with a detailed, descriptive name) in a specific 
folder for your project. Note that to play the video, you must have the 
DivX codec installed (available on the desktop of the Observer 
computer, or at: www.divx.com) 
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Recording a Session (Field) 

See Appendix D: Portable Observation Laboratory Embarkation List for a 
detailed list of the equipment that comprises the portable configuration. 

Digital camcorders (handheld or tripod mounted) and digital voice recorders are 
used to record video and interviews. Firewire and USB connections are used to 
transfer video and audio files to the computer. 

Noldus Observer may be fully used on the laptop only if the USB HASP key is 
attached. 

In all other respects, the process of recording a session in the field is similar to 
that in the laboratory. 

Some notes: 

Take copious notes about what was recorded on each video tape and audio 
recorder, and exactly when. Each time a video tape is started or stopped, the 
time and a brief explanation should be recorded on the tape. 

It can be painstakingly difficult to synchronize multiple simultaneous camera 
angles of the same event after the fact (for example, to set up a picture-in-picture 
shot, which would require editing with Adobe Premiere Pro). 

It is much easier to practice a session once (or several times) before recording, 
and get the video shoot right the first time, than to do a lot of editing the video 
later. 
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Scoring a Session 


“Scoring” or “coding” is the process of turning an observation session (usually a 
video file) into a data file of events, event times, and (optionally) comments. In 
traditional usability testing, this may be done with stopwatches and logbooks, 
which is tedious and prone to error. 


The Observer 5.0 software, by Noldus Information Technology of the 
Netherlands, will be used to assist in coding the observations and analyzing the 
results. Observer is capable of a great deal of customization to deal with different 
types of research (such as time-lapse recording of animals), different hardware 
(such as “old-fashioned” video tape time-code generators), and individual 
preferences. In this guide, we will present many settings (without further 
explanation) that are either specific to the NPS AWE Center, or are the simplest 
for the research conducted here. Advanced users should read Appendix A: 
Advanced Capabilities to learn about more sophisticated configurations, and 
about the simplifications used for this basic procedure. 


Full details are available in the Noldus The Observer manual. The built-in help 
files in The Observer are easily readable, contextually appropriate, and highly 
recommended. Noldus offers telephone and e-mail technical support that has 
also proved to be very helpful. 


The Observer can be installed on multiple computers, but requires the presence 
of the USB HASP key to be fully operational. Without it, it will operate in a 
restricted mode that does not allow changes to configurations, or the use of video 
files. 


Definitions 

The step-by-step tutorial in this section has examples of each of these 
definitions. 


Independent Variables Independent Variables (IVs) represent specific 

information about the participants and the scenario 
in any particular observation session. If you don't 
use tVs. and store all your participant / scenario 
information elsewhere, you lose a lot of analysis 
ca pa bi I i ty b u 1 1 t-i n to Observ e r 
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Behaviors 

Events that are recorded during a scenario are 
called behaviors. There are event behaviors, 
which occur at a single instant in time, and state 
behaviors, which have a duration (see Appendix 

A: Advanced Capabilities, for more details). 
Behaviors are grouped into Behaviorai Ciasses. 

Subjects 

The roles (“billets” or "actors") played by the 
people simultaneously participating in an 
observation session, such as “Team Leader” and 
“Rifleman”. Very often there will be only a single 
subject (the person using the equipment being 
tested), and so subjects will not be set up in 
Observer. 

Scoring or Coding 

The process of turning observed behaviors into a 
data file for later analysis. 

Projects 

A project file contains the configuration, all 
observation and data files, and analysis for a given 
experiment. You can only do analysis on data 
within the same project. 

Workspace 

An Observer workspace is a way to organize what 
projects are visible. Use different workspaces for 
different researchers, or completely different 
research. The Ojbse/ver system in the AWE center 
has a Sample Workspace, a workspace for Class 
Projects, and separate workspaces for theses and 
other research. 

Configuration 

A configuration should not be changed after 
starting to collect data. A configuration is locked 
after data files are made. If you unlock the 
configuration, you must remove the data files, 
make changes to the configuration, and then add 
the data files back. 
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Noldus Observer step-by-step tutorial 
1. Configure The Observer 

See chaplefs 4 arrd 5 of The Observer Reference Manud. We will build a configuration that ends up' looking someifiing 
lihe this: 



You will oily need to do this long configuration process once for your project You shoJd plan on not making changes 
to Ihe configuration after you have started collecting data. Its possible, but likely to cause problems 

1.1. Close other applications and launch the application: The observer 5.0 

1.2. File > New Works|>ace [if a newworitspace will be used) or FllOOpen Workspace. 

File>Workspace> Properties will give you some information, and allow you to Change the 
name of the workspace. 

1.3. File>New Project Enter a descriptive file name and a detailed description Oick [Save]. 


O 


AWE Observation Laboratory Basic User Guide 
Coding a Session 


page 13 


101 































































1.4. Configuration>Sectrfigs„ Set as shown below: 



1 5. Configurationalndependent Variables. Adjust the window and the column 
sizes so that you can see all the columns. Enter information relevant to 
the specific observation session. This information wifi be used later in 
data analysis. Set the Type to Numeric or Nominal, as appropriate. The 

"more info" button with three little dots 3 in the Values column can be 
used to restrict the entry to specific values, to prevent errors (such as 
spelling) iater on. At the very least, you will usually have an Independent 
Variable (IV) to identify the specific participant being observed, and an IV 
to identify the researching scoring the session. Having too many I Vs can 
be slightly tedious to enter, but wili cause no other problems. During later 
analysis, you may regret having recorded too few I Vs. _ 

Q otmo Project - lndtpend«nl VariaM«& 
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1 6. Configuration >Behavioral Clashes. [Adjust the wiiTdowas necessary) 

1.6.1. The Observer calls any event that you observe a Behavior. Behaviors 
are grouped logicaliy into similar Behavioral Classes. Enter your 
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1.6.2. Enter Behaviors as elements within each Behavioral Class. Clicking on 

the more info button 3 in the Elements column allows you to enter the 
Behaviors in each class. 

Leave Modifier Classes set tO None. 

Always allow The Observer \o set the behavior code for you. 


Change Properties to Event for instantaneous events^ such as error 
behaviors. This step is easy to missl 
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1.6.3. You should use State Behaviors for scoring events that have a specified 
duration, such as time to complete a task. The participant must always be 
in one of the states in every state behavior, such as: ON Task 1, OFF Task 1. 
This is easier to work with than separate events like START Task 1, END 
Task 1, because The Observer will automatically calculate the duration you 
are in a task, error condition, etc. 

1.7. Configuration>Check Configuration. The Observer will check your configuration for 
connpleteness and conflicts. 

1.8. Configuration>Review Configuration. File> Print. Printacopy of your configuration for use 
during scoring. 
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2. Starl an observatiorN session 



will up for in obsin/itio n s^gsion {from a imdia fill ricordid earlier) 


2 . 1 . Launch and configure the ObservaliDn Modute: Ob^rvation> Start 

Observing. Yqlj will only need \o do this bng conlig uration process once 
for your project. Scoring ihe videos wH! proceed much more quickly after 
the fnst Urre. 

2 ^.^. if you Ihinlt Ihe Observation Module settings ane incorrect larsome 
reason, you can use Customize>Reset Custom Izalion {to set 
everything to the defauHs) before setting the custom izatiaus below. 
The Help button in each of the following wi ndows ha s excellent 
descriptions of these settings 

2.1.2. Vpew>Codes, yonitoi. Timers, and Video Control should be 

checked, others are cpiional. (Monitoi arb Video Controt may not be 
available- until Tiining is sat in step 2.1.4). View^System Status ran 
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be useful la display Independe nt Variables or other intdmialjon. as 
shown ir^th* screen snot above. 


2.1,3. Arrange and the wmdovv^ conveniently, ae shown above 


2.1.4. Cu£E]Dmt 2 e>T]tnlno. Set as shown below. You should atways use Ibe 


^pnc Timing fer a II ob^scrv^ly ng 


TimiRt 


Evvii frTTTTQ^ ScHid an 


C livfl i;^0^TwtirOK4 


; 1IM ifaNi/ltm 
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0&s9fV3r*3n. y<xj csn nave prowsms wm 
cQcies ypii 5&rT m (he mjdScJle e 

vidsQ end ihsn rg^mid ff p-asr ‘iera" 


2.1 .S. Cu£bDmllza>dperat}ofl. Sel as shown below; 



1 


A.WE Observatfeen Lob^r^ry Bmac. User OLildb 
A £#ul&n 


P*B»17 













2 , 1 , 6 , Cti*tcmiit>windoiiif fawtrttw>Coiiiti, Sel stiown below 



2 . 1 , 7 . Cusioinlze>Wln<lowfroi]Cfttei>f¥Hit log Set a s stiown tie low 


Evmjt Lof: lhF0j^lh« 


CiAnr^ 
P* SrtmTTiw 


Obfjiay Sonu Tiinv Ax 
r E}«iW^ wnJfl 

>ip Tr^4 nvi mm li <M1 

Tp«^ (hh rm 11,4111 




Ttn»' flptoft 
ia Littjng 

aniAw 


2A . 8, Cu*tembe>Wfadowr frB|iwtfei>Tlmtri S et gs Sh-own befw 


TlriHri Prsp^irlrl^i 


T4-»iii-D^ iploiwi] 

r ' g^dTn« 

■flj Tini>» 

T*ni P^DUiUtH 

|rr^ 3 




f Tifftll 

r Sam^p^T m* 

P Tfn^k ^^«xl Safripht 


Carvrl 1 


bMn 


2.1.9. Cu&t)afnl^€>W1 ndow Prvpcftles>VldEo Control. &Bno EXPANDED- 
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2.2. Cu5tQmize>Data stomqe. Set as shown below: 



2.3. nie>Open> Media File. If this is unavalable, check the Timing selting in step 214 

2.4. Data>Stait Observation. You will star! an obseivalion to score each separate observation [that is, a 
recorded video). The Observer will automatically supply a name [because of the Auto-Name setting in 
CustomlzoData StOfageT A Title allows you to enter a more d etailed description 



2.5. Independent Variables. Enter the values of the IVs for this session. 

2.6. Finally, clicking this start butto n will start the observation session: 



2.6.1. As the video plays, score behaviors by clicking the appropriate box 
in the Codes window. Use the Video Control window to pause or rewind 
the video. 


O 
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2B.1. To entera oommenl double-cSIek in the commt^ti. column of the 
Event Loo You onust press lEntetl when done typing a comment. 

2.S.3. To 0 drt a scored event: 

2-&.3,1 Pau^ the video using, the video Control window. 

2.6.3.2. [>&ca>Edit QbceivaiiQri. and edit the event (eg as necessary. 

2.5.3.3. &atai> unch&ck Exiit ObserraNon to rstum Eo scoring mCHle. 
2.M. When scoring the video is oorrplete. choose: [ka>£nd OtweivatiDn 

2.7. To go track and edil an exisEing observation file after ending an 

□bservalioni choose [Mu>sartobservAtlcn (as in step 2.4) and enler the 

name ef the ewisti og obseivatien file -(Phsteatf ^f a new file name). If 

prompted For "Media Fite Position", do norf check "Use Time in Media Fiel 

as-Start Time tor Data Fife”. 



Continue staming the observation normailv. You can add new events to 
the previous file, or go inlo Edit Observation mode to ediles^isting events. 

2.8. After you have completed scoring all the videos, close the 

Module and return lo the Mam M odule: Fiioneturn to hum. VMien 
prompted to Sive the oustomiKatlon fide, dick [0»q. The custonnization 
saves ygur pbservation Module settings, window arren^ments, etp. 
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Analyzing Data 

Yoti can analyze data collected using No Id us Otmnmrin two ^ne;ral ways; 
exporting tm observations into a dedicated sratisficis pacKage sdctv as SPSS. S- 
Plus, or Excel, or you can use Tibe O^serve/'s built-in basic analysis tools You 
can also th»n axport thal analysis to another pnagram. For most daEa analysis, 
you will need to consul! ihe docuimenrtanor^ abo ut (he specific ways you want to 
organize and analyze your data 

fxporfjn^ data from The Observer 

Th$ OfrseA^r rsw files are normally foynd in ine dir^ry: 

Ci\DocuFwwnt5 and Setting i\All U»n\hloldu(\iThe ObserverVWork* fwiees^ Prodwt^X Mcwr 
f^nscr Fills 

You stiDuld not move data fileefor an'y other fiCes} from IheirdefaLit Idcalion^. 
Data riles have ine .odf (lie extension (example: DcnHOOOi.odl^, bin are normal 
text files. Cop/the deia flies you want to another locaiion (IlKe a folder on your 
desirtop), artd rename them vrith a ,ort exiensign, They now gsn pe opened with 

Nolepad, Word, etc. Do not edit or alEer tfie onginal _odf files. 

It is probably easier to use Observer for organizing the taw data and performing 
basic stalislics (see the next sectionJ before exporting it lo another program. 

Using The Observer's analysis toots 

Dsla snalysiSi in T^s Observer is a muni-step process. First, you select the data 
to analyze, and then you can view the data graphically Wsw), or 

perform elementary statistics or reliability analysis on it. 


1. Elemen la ry Statist ics. 

To underhand alhhe options in this lundion, try generating a r^urnber ot 
ditfereni slatlstics shieels to see the difTsreoces. 

1.1. F rom Tlw Ohae-rver Main Module (rot the Observation module — see step 
2 .2 above); Anifty&ln > Eleirrantary SblisU cs > Open Afialysll Pmflli... 

This will edit the currenl Slatls^ics Profile, which defines what statistics will 
be computed, a nd how Ihe output will look. You can create New Analysts 
Profiles and save different settings In each ^Use 4ctive Ditm Profllc;> is flr^e 
for Ebe finsi use, If you dont already have one. The Observer will create a 
generic da^ pr^flte for you that Includes aif the da(a collected (usually 
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what you want). 



1 2. On the Statistics tab, choose the statistics you are interested in. Durations 
will only apply if you have used state behaviors or have defined time 
intervais between certain events. 

1.3. On the Layout tab, turn off the Modifiers and the Subjea, since you will 
normally not use either. 

Use the up/down and lefl/right arrows to change the arrangement of rows 
and columns on the output statistics. You should probably run some 
different layouts to figure out how you want the Statistics Results to look. 
Turn on Interval only if you are using state behaviors, or have defined time 
intervals between certain events. 

Turn on any independent Variable values you would like to show on the 
Statistic Result output. 

Change sheet Per to <one sheet> to put all the data on a single sheet, 
instead of on separate tabs. Otherwise, choose how you want the tabs 
broken out. 
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1.4. Click Run to create the Statistics Result from this Analysis Profile. Note that 
you can select cells (using shift+ click to select multiple rows) in the 
Statistic Results output and paste them directly into Excef and other 
applications. 
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2 . Time-Event View. A Time-Event Profile is a quick way to look at data 
graphically, or to organize it to export to other software. 


2.1. Use the Layout Tab to choose a Time-Event Table or Time-Event Plot. In the 
examples below, sheet per is set to <one sheet> so that aii data is shown 
on a single plot or tabie. Ciick Ryn to create the tabie or plot. 




Tim«-Evftnl Vi#w 


Date Inpu) Layovt | Dqjq Output | 

r Tinn^Cvefitleble 
fi' plme-EveniiEllcHi 


Pa^et I 

[ Bun i 


Sortordet 

£olutTirt$ 


Pnnnly 


F Dj^'q ^Chr 

FObe^t^sTian 

nSubJftet 


£hee( psr 

]<one 3heet> 3 


^iDsa I 

Canealj 

Help I 


r~ Merge Oata fei anfi imeival 
f” MfirfiP hP*''‘.-"Srir/minditiir - 


2 2. Sample Time-Event Plot You may have to resize the window to see all the 
data. 
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2.3. Sample Time-Event Table. This data can be cut and pasted into other 
Windows appiications> or exported directiy to .XLS o r .CSV fiies. 


^ Demo Project * Time-Event View Time-Eve. 3 QQ 

Start TTm«| Behavioral Class 

1 B«hav]or 

lobservationj 

1 IF 


gvenit 

DemoOCul 




PemoOTOl 


2 32 

Evanls 

Bv^nlt 

DemoDCEil 


S72 

Events 

everilt 

DennoaOl 

— 

s.eo 

Everns 

e^nU 

bemg0002 


6.DQ 

Eva nits 


DsmoDOP 


B.oa 


eirenlT 

bemoDCO^ 


e.24 

Eventis 

ewriU 

DefnoOOGil 


6,38 

Events 

ewenl2 

OemcjDOO^ 


6,76 

Events 

B’ltenn 

DsitioDOCB 


0.2B 

Emors 

Wrong dctron 

DemoDCbl 


< 1 v[\AII Data Date Profiled All * [ 

1 

*ri 
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3.. DataSfilKtm. 

Dald PnQfir^g r^s^rict q [ llPter ^ur an^ly&is V.o 0-f the 

daEa. 

3.1. From The Cb^Brv^r Main (not th# Observation modul-e - -se^ s4^p 

2 2 above}; DatA > Data ProFlie > Hew... Give the data pnofike a name. 
Note that a data prone window has lEems in rov^, and si^c tabs along the 
bottom: 

DviM P^oJ*Et -■ Pf^oAfli II) - CkT-i 'Srii-^IlQn 
Irtdflfr* VMljIts; [PfcleS^ 3 




3IiKln*p*n thini Variabl i i 2 ^6*was 1 &Bhart(K? j[ Msd ftflfs I Iriw^ f\ 

3.2. Edri a row on the appropriate lab to creaie Ihe Ulter for your data. 

3.3. Once youVe created a fitter lor your data using a PaEa Pr-gHle. you can 
select that Data Pvdftte in Eremenury STatl^tk^, Hme-Eveiti Piol. end other 

ar^ly^is funclions. 
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Video Equipment: Other Capabilities 

The video equipment in the observation laboratory can be used for a variety of 
other functions, including: 

• Capturing VHS video tapes to computer files or to DVD. 

• Recording (“burning”) computer media files onto DVDs that can be 
watched on standard home or computer DVD players. 

• Recording training sessions from multiple video angles and direct screen 
capture to computer media files. 

Capturing a VHS video tape to a computer file 

1. Complete the standard system start-up 

2. Put the videotape to be recorded from into the VCR. Rewind the tape to the 
starting position. 

3. The output cables from the VCR go to ‘video and audio input #4’ on the 
Video Mixer. The audio output of the video mixer gets plugged directly into 
the Canopus board on the CPU. You will have to disconnect the current 
audio inputs in order to do this. Note what they were so you can put them 
back again when you are finished. 

4. Set Video Mixer to input source D, which corresponds to input #4. If the video 
input is not displayed, see the Focus MX-4 mixer manual to check the video 
“Route” setting for input 4. 

6. On the computer, launch the Canopus Media Cruise application. 

6. Set the Canopus Media Cruise to video recorder mode. 

7. Click “Format”, and set it to “MPEG2” and “Program Stream”. 

8. Click “Parameter”: enable and enter the max video capture time (i.e. the 
playing time of the video tape.) You can stop the recording earlier, but it will 
run all night if you don’t put in a time. 

9. Click the folder icon on Canopus; this is also known as the “capture File 
Configuration". Enter a file name and location. 

10. Press Play on the VCR to preview the audio and video levels. When these 
are satisfactory, rewind the video to its starting position. 

11. On the Canopus control panel on your computer screen, click the button with 
the red dot to begin recording, then immediately press play on the VCR. 
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12. Click the button with the white square to stop the recording. 


Burnins video files to DVD 

1. Launch Sonic My-DVD Studio Deiuxe Suite. 

2. Choose Video, then Create Project, then DVD. 

3. Click the “Add fiies” button, then select the file that you want to burn onto a 
DVD. 

4. Change the background and text as desired. This is the time to add subtitles, 
trim or crop, etc. 

5. The quality setting should be set to “Fit to DVD”. 

6. Insert a recordable DVD into the computer. (Note: the CPU in the 
Observation room only works with ‘DVD plus’ discs, such as ‘DVD +R’ or 
‘DVD +RW’. It will not work with ‘DVD dash’ discs, such as ‘DVD-R’ or ‘DVD- 
RW’. 

7. To create the DVD, click on the big red “Burn” button. 
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Appendix A. Observer: Advanced Capabilities 

The following are the main restrictions that have been used in this Noldus The 
Observer 5.0 configuration to simplify the tutorial in this guide: 

• Only event behaviors - no state behaviors 

• One st/b/ecf only (and so, no channels) 

• No modifiers 

• No live scoring - scoring from video only 

• No use of the event-logging keyboard 

• Recording video using only Camtasia and Camera Viewer, rather than the 
Canopus MPEG-2 encoder card. 

The Observer \s capable of much more sophisticated configurations. More in- 
depth research will benefit from using features such as: 

State behaviors are useful for coding tasks in usability analysis. State behaviors 
have a duration, while event behaviors are single points in time. State behaviors 
in a behavior class (e.g. "posture") are mutually exclusive and must define all 
possibilities (such as: standing, sitting, prone, other). Every subject must be in 
some specific state behavior in each state behavior class at all times. Some 
usability researchers will use states to identify tasks being performed. Analysis 
on the duration of events is easier when using state behaviors than with event 
behaviors. 

You can coerce two event behaviors to act like the beginning and end of a state 
behavior, to get data on duration, but it's easier to just use state behaviors. 

Multiple subjects. Subjects are the roles played by the participants, such as 
"Primary User," and "Assistant". Subjects would only be used if there was more 
than one person simultaneously participating in a scenario. 

Modifiers can be used to refine behaviors, such as a "walking" behavior, with 
modifiers like "slowly" and "quickly", rather than many separate individual 
behaviors, such as "walking quickly" and "walking slowly". Using modifiers allows 
for more options during analysis to group and categorize the data. 

You can code behavior live while it is being recorded, and then go back to the 
video file and refine the coding later. This might save time for experienced coders 
doing a lot of observations, but is generally too much to do at once for beginners. 

The event-logging keyboard can be customized to the specific events used, and 
labels attached to the keys. The jog-shuttle dial, used to control video clips, is 
extremely helpful. There is a "programming switch" on the upper right side of the 
X-Keys keyboard that should show red to program, black to use. You may have 
to exit other applications when programming the keyboard, and sometimes even 
restart the computer to get the macros to “stick”. 
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Event Summaries (NoEdus The Observer Reference f^Januaf section S.4) can be 
rielp^] for generating quick re ports^. 

Reliability ana^si&-consistency between dlfferenicodtrsKor a euigte coder 

s^fifig ihe same video 

Lag-sequential analysis -- identifying events that tend to IqIIduv ar precede other 
events, and even "chains" of events. 

The Canopus MPEG 2 encoder card is a faster, higher^iuair^ way to record 
video if onfy wcfeo from the cameras is reqm/ed The Canopus system ^card and 
Medi^orurse applicatioo) record events oi^ the screen, suoh as ihO; WC 
screen. 


1. Co nflgu raLlan> Subjects 

SutfKts wm only be used if the€e spe mufltste people riterectmg ^ s scenano si once 

wa often not be ysed If tnere ts crily ofhe wJCitecr. fw cgnflsyrgcian ts necesMiy ror 
If subjects 3 re used, ihe of the -rcile. pcertiun. or bin^t thev ere 

perforrnng m E he scenarif> If ne ceBsaiy. Ihe indivfdLBl pariKi pant's actual name should be 
sEored a& independe^nt varables, not as ihe sufatect name _ 


• Ti- 

piPjKl ■ -tu^^lE 



1 

1 rc-i-l| 

1 



> 




iHrin 

^ tj- infcw I ■ 

* * 



m 


1 1. ConflQuratlEiin^Clunnils-. Chann^Fe are required if there i& more then ana 
sublet. In your scenario, subjects may be eltowed to perforin ell defined 
behaviors, or, by limiting the available channels, subjects may be 
restricted to anly specified behavior desses. TTiisis mainly to prevent 
data input errors, and group subjects and behaviors more logically. 


Topics to ignore Ir^ using Noldus The Ofesorver 

1. Codf^g from vfdeo/epe. In the past, analog videotape recorders with lime- 
code generators" were used in beheviioral research. Muctiof the video and 
timing configuralionin The Otisewer 5.0 deal with these situations, and 
should be ignored. 

2. ht&n/si (sietes or e¥&rtts) rBCorq^og. Situalions calling for these types of 
recording methods are uncommon. As opposed to conUrruo^s recording 
(which we use), Ihese generally apply Ic tirne-lapse or long-le rm 
observations. 

For more irformalion on Ihese capabililiis. see thi Naldus Th§ QbsmFV§r 5.0 
docuETientation. Also. Noldos has been very helpful in email and telephone tech 
support. 
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Appendix B. Observer Lab Components 


OiseA'e/ :sr<j£/o-rr components 


Observer worKststion 

computer 

TOOB HD. rOB f^AM. Pmf^ DVD^t CCf^Rw^vD+RW. 
f ATI R^d&Cf} XJOO Senas 120iWS 

iwtuspKial hard-vife andicritvvare ctescnbed ber&w 

Event logging keyboard 

Pf Engineering .S-keys S£ 

Camera joy^ick control 

TsWirntnCS CP-iJV- VC4 

Video miuer 

FociMSEn!wcmm$MX-4 

Srik&MS. %ucod<1fli mihyah tivA fIbmwirVi iSiDwnitrtih ■[ 

t^xt Iwkiifn'o boin 

mlniDV VCR ^ponable) 

SonyGV-DICCO 

Wireless micnophone 
receiver 

AsniengOOR 

Mobile device camefa 

Kjsm GigsiysiBirr iRX'Sofmm cornpany 

receiver 


Observer microphorte 

£fruraWX-i#T2£MC 

Mic amp 


Spe-aker amp 

US AuCKs i^rirfwvndrP--T2 

Speakers 

Radio Sh3ck400-4m 


Showths pfivrsw aM iwogbam &LJlpute er (he video muter 

Qn^-way mirrored gla$$ 

4S‘>C36' 


Observer workstation (specM hardware and software) 


Noldus HASP USB serial key 

Mlim be present lo use iskJWvft Observer soll^i^afe 

r^mtur OB«0-»4J2- 

OiSfl^ 


KokJus The Observer v5.0 

opprcatten (or coding and recording 

Pen^iof 

Canopus MPEG Pro 
MVRIOQflSX PCI card 

Video encoder Iward ■ feai-time MPEO-2 encoder 
^frQ^^ aralpg or DV inpul) TalrEsThe a&TtjarDtJfpijI: of 

INe video mxer CirKipui ■» m Sbt jdh; 44ddj 4Sat 

Mhi^i ind iL^ciDrt H: ME .^ftra«_tiniBUi.ui 

Canopus Mediacrui&e 

Progiram loencode- analogv*icleo rr^ut to MPEG-2 
bin 

Canopus MVR DV-MPEG Fi^e 

Used tdcofwerr r^w DV flies tc MI=£G^2 fdimat 


a AVfB Ofa&ervatfeon LBb«r^rv U*&t □ uldc- 31 

B. Lab C6rr\^^^n^ 


119 




























Converter 


Euresys Picolo PCI card 

Video capture board (PCI slot 2). Allows screen 
capture of the program output from the video mixer 
(the Canopus board does not). 

Noldus Camera Viewer 

Viewer for preview video in to Picolo Card from video 
mixer 

Sonic MyDVD Deluxe Suite 
6.1 

Software DVD player/recorder. Watch, edit, and burn 
DVDs (cannot use PowerDVD or WinDVD - they 
conflict with The Observer) 

Techsmith Camtasia Studio 
2.0.3 

Record, edit, publish videos of PC desktop activity 

RealVNC 4.0 (in viewer mode) Observe / control the participant’s computer screen 

X-keys Macro Works 

Controls the event logging keyboard 

DivX codec 5.05, DivXplayer Video codec and player for highly compressed DivX 

2 0 video files 

Adobe Premier Pro 1.5 

Professional video editing software 

Participant station components 

Participant workstation 
computer 

Windows 2000, 512MB RAM. 2Ghz Pentium 4. Two 18 GB 
drives. Could be any computer, but must run RealVNC 

Server (which requires administrator access to install) 

Pan / tilt / zoom cameras 
(2) 

Canon VC-C4 and Canon VC-C50i 

Ceiling microphone 

Shure MX200 omnidirectional 

Wireless microphone 

Azden 31 LT freq: A4 171.905 Mhz 

Mobile device camera 

Noldus MDC-03N 

Office furniture 

Desk, conference table, dry-erase white board, chairs 

Individual military 
equipment 

Helmet, flak jacket, MOLLE harness and cartridge belt, 
rubber M-16s 
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Appendix C. Observation Lab Technical Reference 

The- mast commonly refenenoed bps and trfluWBshooting facte are Nghlighted. 


Observer Station 

Observer worksteti<Mi 
cornpute-r 



Cofnpulgr nsmg; D21ISIM51 (jkiI: dh Mras dorrsml' 


IP Ad^fflSS- CSlsrtic): 
$UM. 


Galjev^: 

DNS 

MAC AddT0B&: 

Dgt Assel Tig; 
AdminisIr^Qf passvrord: 
normal Liewnsme: 


13l.i:0,14?.11. 
£S&.^Si.£S2.0 
131.120.,14l. 1 

ni.120.1^40 
131,120.16,41 
aD-ai-ll-3D“D2-22 

MOLduj£M^I 
WDldtaj Observer 


paBwcrd: 

6aci4 6<i4ip (ChoQl ima^a) by jHn S££-24^, on 

IS Fab 05. Image rtas scored on: 


Camera joystick control 



A h^tk i| IIIW 4F thfr 

c&rnBfa& ia Eumed off. or olhaf pfo blefnc; occur. 


Fifsl, prs«s Ihe button ’Oarraspcfiding to lha- camars 
you wanl Id control , lhan use ijfta jeyciiclc, zoom, of 
loculi donlroir, tbcfo i% a brivf dol^ befom the 
eamets will begin le move. 


Hob ihe^i b^iiron? dchwi Icir 4 je^dnd* t&eai 

(dirwiLjn, fwui. flAd 

mighrl want Eo lum Ihe aiiflo^Dcus OFF, so dial 'i 

doesn'l hieep ciiangng. 


Video mixer Focus MX-4 There ara alwaya two wWao eourceaaelecled fwilh the 

two nowa of tMitlcms at tha to p of Ifie miK ei) - lha 
"ewfenr (or "cuT) couroe. and Iho- 'hokI" sooroo. 


CoriTIgurtiig vkleo inputs 

If /ou change the inpvl ^ «|e (lab cameras. 

MDC, VCR, atc.J. you must seE Iheni up by preacing 
lha [ROUTE] bullon <ch 2 in manual^, hteta v^ihelher 
Ini^ite uso Itio S^iioo Of Iho RCA-^la 
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wnsuic «wij3! rori!^ copwsrte 

PfcOierrmyou e!(penen« 


V w pp n tUBp^3y two inpijB 

sonutlaffieauBty (ev^niTiOLJ^ ChBre 
A rp rou r rtpot*) 


Useful effesas rndude p*ctxjra-in- 
pHCtxjre {P*P), and 

wte^bv-sKie PUS 


C-dnflQfjrhg W«& ou|(Hfls 

Press Ehe [Dl SPiLAYl bi^n tini^ % S^ 

dirr»erri vrewt or th« camera »led»orm on Ehe 
PREVIEW moralor. whch n uied fv conihgiuiilfin. 
rine ad 113? output to the compuiif rs ihtonwi cm the 

PROGRAtAimnbar 


B-Bskc oper^an ^ da Chia fu^l 


By Bdjustiog Ihe camerM' zoo^ fewel 
and po^EKjf^, snd rigw a 

cut ad^fustEd. VQU can anan^ 
muttipie vww^ of Ihe partKipant room 
pretty much Ihowertf yoij warn For 
a Kampie. ^ v^u w^nt to ria\r€i 1*^0 
vftews a} Ehe pariici pant next \n b 
VNC windcw 


1. Scitfie OtBcrver CamEraViewEf 
window lo 1 /I size 

2. U&e irgnfi^ioTi fl© (loiMo-bonomj. 
ora PIP window, ta spifl Ihe view 
between the cametras 

3. HKfe tne ngM hart of We 
Camera Viewer wkndow &ehjnd lh$ 
\iFNq window 

A ^ju 9t the oamef ee' zoom and pa n 
ip show the views pr the peniapeni! 
that you want 


Slide EtitT-Baf U3 ih# lop foil and ctfitif dw 
jovstck. SeiKf vouf fiffo inpul*3 [oufrenl sncl n«rt). by 
T^pptt^ o«ne cvdor In ihetep rpw oT ^URq&a $nd 
one in t he bottom rpw 


Side-b^-side c ut 

Press (Ine TR_ANaiTIONS [BASICI budofi 

Prtss Ehe awow to niww th§ du rEcr to traneitkan 

fl7 (s+da-bir-ticSe cots) Press OK 

Move the T-Bgriij and down |q njn (he EianaiDon 
Ad^isl the cameras if simoteneous eide^ side viewE 
aredesir^ (with the T-6ar hatf-wayH 


Picturv-tn-pidlure (PlPf 

The "next' wjroe wEl be the Bmall PP window 

Pf ess (he p?ps butlon 

Use the T-fla r to si^ e Hie PIP window 

Use th& joyaick b move the PaPwtfi*>w into position 


Press (PLAY] io slerl the PIP on Ehe ftogram oiipul 
S* careful not lo bunp the joyUick or T-bar while 

fficordogr 


To compietety reset Hie mixer ^including finnwarie) 
beck Eo factory state power on wtile holdng down 
Shift and Setup (mantel Ch13} You wll may have In 
settq3 ihe ihput [ROUTE] 


LMd u^rsdies vis Emerwt 

IP Address I3l.l20.l4v.llt (slalbc] 

SNM 25 5.255.Z52.D 

geteway. 131 .12C. 144.1 
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MAC Address: 00“02-ea“4c“47“e6 

sm 050518 

Check FOCUS website for firmware upgrades 


miniDV VCR 
(portable) 



Can run off battery or A/C power. Needs special A/V 
input cable (3 RCA F to 1/8'' plug) 

Can convert analog (RCA or S-Videc) input to raw DV 
over firewire for capture to computer useful with 
laptop, but the Canopus MPEG-2 encoding board is 
preferred when available (on desktop computer only). 


Wireless microphone receiver 



Power ON, check antenna 
Adjust volume as required 


Mobile device camera receiver Power on, check antenna. 



Will suffer RF interference from wireless networking 
equipment. 


Observer microphone 



Push To Talk (PTT) “touch and hold to talk Can 
change PTT function to "always on," or other settings, 
by setting DIP switches (see mic manual). 


Mic amp PreSonus MP20 CHI: Ceiling mic in Participanrt room 

CH2: PTT mic in Observer room 
Both channels: 

ON (lit): 0, +48V, 60Ha, L7R OFF (unlit): PAD 
GAIN: as desire IDS: max PAN: center 



M aste r Level: as desir^ Phones: as desired 
Adfust GAIN to control a mic's recorded volume. 
Speaker amp CHI: Ceilirg speaker in Participant room 

CH2: Ceiling speaker in Observer room “ note that 
this is completely separate from the computer 
speakers, which may (or may not) be playing the 
same audio 


O 
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headphones Cff (oul). ^po*ome aisout a on eh i 12 


Obsefver/ParticipanI ncKjm Ensure 

Speakers cow^iwnf m gs^ 

Mo£e lhat these are t)ola% Eeparate f ram She oompuler 

skaters 


Noms HASP USB serial key 

LifiSflifi nurrtb*r OBCr5aa5422.DGAA 



^ vflty careful h rt ti-iB tt expenwv# to 
rf 

DO NOT tpdasedrrveFS Td^ the Aladdin USB HASP, il 

can caijg« your utss hasp key (doiQle) and Meum 

lo^Op#orfci»^ Tecofred rt 

1 Dd#nbad the latest rSommancFfcrTe HASP ^ drivers 

from 

nitp Biaddm 

arsp and save tTiem to a footer svch C ^Emp 

2 FieniiheVVinddw* Start irariu.telicl Rui^ and theri 
type c«>d {a commatid M^indoiy win open) 

3 Type cd citiemp 

4 P^sroovtarry har<f*arei«7iffom^USBof LP^ 

(5Qrt5 

5 Type hinstaH /r <[te ftrttove exifilng HASP drr^virt) 

$ Type iilnstad fi install Ihe new di^verSi} 

7 Close (he wfindcMv and "iBert HASP iwy 


NbUus TTie Obsen^ 

LK»rw# num&ef: OB050-06422-DC5 j^A 

n^ial^Ad udpfrfa B^rtrin 

MsKay Ka«hiT»tugi 

Updated to vSCSt'lrom 


IDJI 

iir-f 



~Sda«i and inuAJitfb^t^ ~ 

Johi^ J McGraw. Sam Ernginw, j rrcgro^nohhJt com 

&Bfl vin ^ael^et tn^nical rep (rwn Ihe Nelhed aixlsi. 

B v-inR0«5wl4^Noldua< ocm 

Ntfdift I ntofmaiiw Tecftnolpo^v ine. 

75i Miller Ojwe 

St4#-E-& 

Le»lwr-j VA 20i75 

Phgfie 703.771.044a(8CO-m&>31 
fsn T03.771-DW1 

t49bia he^OnK. normilwficf^iig tram S3an(iard 

tMduA Techr^p^ Sup^rl Krw^^edgefidH 
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I4clcl|^» »vpp<r1 rflqgw^ fOfrf\ 
liE1p:Y/YAm fi QHU& tiDmffi4lf/{la;2fXHQ?0^ 


C&nopus MPE3 Pro 
MVPK^QSX PCI wrfl 



IrtlEmal PCI card - haiTj^-are MPEGr-2 eticodBr/ 
decod&r Ocntrcllcd bf the Cancput MEdaCiuiH 
applcatrofl 


Ca no pus Cfu^se VkJeo rwofder wPlware Tar the Canopys card. Ueed 

io encode j he coipu! or the video rTHwer {arv3 

lh« audio inputs connected direct)/ to lh« caid;^ to 
MPEGl (smalief3 or MPEG:^ (reaxnnendw^ ftle* 

Tbifi IE Ehe hgnest riualrtv way to leDord! vKleo from 
Ihe video cameraB Qrty ff jrau neadCafFtpc/oracrdsn 
too. you mijsf ltw c*>iMira aod Jiha Ftooto 
Carwrsv^&i^ier- ^p^^siwa fbf wdso n^puf 

VHSeoCDcMiity<MPEGi. upeftteftEShsion)* 
abocK lUMsmiin (irw - 650WB = i CD) 

DVD quality ^MP6G2. M2P file eKiemicn) ±5 gbout 
40 HHrnin i [2 hr - 4 fl GB = ^ DVD-) 

The Medflcruis* window bypa**** Window* 
ope^aDng system and carmot te captuted vnini>aeen 
slyjts or Carrtiasia 

Befone recooding. iel the TORMAT VCD MPEGi fof 

Ihe Bjnallest files, or DVD MPEGS for ht^mr quBlity, 

BAfere recoding, eel im FILE to t«»rd to touer 

*cwi), 

can alBo t# uted to piay MPEGr-2 video. 



Uhsompressed Csv files ftonn a camcorder ( AVi 
ejdetisjon) yM ilJffiJ! 200 M B / rr^ n * 11 ij0 i hr 


Canopus MVP DV-MPEG FH& Will corwert a raw DV rile (usuaiy dOwftoaded from a 
Coft\/8Fier camcooierj to MPEG2 (or M PEGI) Takes almost 

feaHirneto ptoces* (a lOmh file 1*k« about 30mri to 
corwert) 


Elacafd i.O MPEiCS-2 video player application t hat wortewrlh 

hfoidws OOserver to p*ay MPeG-2 wdeo wnen sconnp 
vitteds WithouS Ifis prcjgram, MPEG-I pigylJSfJh m 

t^cildLE ObBerver has limited fiivCiDnsIrty 
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Inrlj&mai PCI eertJ - turns aneiofl mit 3 
^194?! stneam Does ncft ericoOe lo MP£G2. twJ. 
(jrtike ine C^rmpus rjanc^. the Pcota C^merflV^ewer 

windm -can be screen caf^ur^ti CantaEia, or [he 
WihdwE opftraltr^ 6y«em 


The ObseTver Camd<s\//ewe< 
ViflM Pflr ihfr 

PimJo pci esfd 


LFsed Ea capture pfrtgfam DLitput of She v»dec nwEi 
with Camtafiia The KHitartte erihatwiiidtiw can be 
captuffrcl by ficrwn thoEa or camiasia 


Chech Command > UvieTo start ^\Oeo. urchecif 
Command > Uva ic rnakecomfigunalion 


Iftiixmanl SAttlitgs^ 

CcfiBg > lnpulCc»fvn9yritlaHiri:h9ekfifii^^^^c^ 

looks 

ifustng flitab S- Vidaa input iaok. S-Vid eo via 
M im j> ^ fy Pg?}, $• Vulfi^ ^2^ L*^t 

if usAng flcBito Bf4C mput /atr^r■ Oomposaevta 
BNC CcmPQsrte NTS^ErA 

^nd Reg[en qf Inten-^f Cli^k 
either' /l’of in mn HtongoirUgl am Vfrrtwal tfieh 
drag window [o^sired siz# 

view > Ahviya Dn Top As desired 

View > Toolbar Off 

View > SEatuE Bar Off 


flUiw tefltngt f^E shflvM not needadputtnwits): 

Ccnflg > Color Fannal: RGG 24 

CcnfIg > Adjuitments Restore defaudi values 
Ai^ifiE during viij«o piBy ^liva) rf desirod 

Ccnfig > Filters a.nd Sellings AllOFFeKce^ Full 
Tange ADC 

Ccnfig > Ajc EiulslElQn Settlngi Frame (Fields are 
graibbed In ar^ order) 

1/ O Op e railDhs Fslo ings atioiAt ns«d to be 

c»Tange^inim*Twnti 

I/O Operations > I/O Lines 4 butbons on top 
enecJflfrd 
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I/O Operations > Trigger Config: Rising edge 
I/O Operations > Arm-Acquisition: unchecked 
I/O Operations > Live-TriggerStop: unchecked 


Sonic MyDVD Studio Deluxe Used to burn DVDs that are viewable in any DVD 

player, with simple menus, etc. Has a very simple 
video editor that can trim and combine clips and add 
titles, but not much else. Will “fit” (transcode) video 
files to the available size on the DVD disk This is rxjt 
the "LE” version that comes with the DVD burner, 
which can not do editing, resizing, etc 


Techsmith Cemtasia Recorder 
2.0.3 



Used to capture (to AVI video file) anything shown on 
the computer screen, including the contents of the 
CameraViewer window, but NOT including the 
Canopus Mediacruise window. 

Suggested settlings: 

View menu: Standard, o/Tcheck ail toolbars 

Effects > An notation: uncheck al I optio ns 

Effect5>Audio: checkRecordAudio(only) 

Effects>Cursor: ShowCursor 

Effects>Zoom & Pan: uncheck all options 

Too Is > 0 pti on s > A VI ta b: uncheck Auto 

Configure and Time-lapse capture. Set to 15.0 
frames/s^ Check: Interleave audio every 1 seconds 


Tools>Options>AVI tab > Video Setups 
Compressor: DivX codec, [OK] 


Tools>Optlons>AVI tab > Audio Setup: Format 
use MPEG Layer-3 if available (for smaller file size), 
otherwise PCM. Attributes: 22 kHz (or more). Mono 
(16-bit, if PCM), [OK] 

You might also check the Windows sound settings: 


Start Menu>Control Panel>Sotinds and Audio 
Devices>Audio Tab, Sound recording: Default 

Device: (Sound card), Volume: ensure that "Line-In" 
is checked, volume to max 


Tools>Optlons>File tab: check Ask for Fite Name 


Tools>Optlons>Flle tab: check After save, play the 
video, u/TCheck After save, edit the video. 


O 
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Techsmith Camtasia Studio A simple video editing application that's good for quick 
2 0 3 trimming of clips, adding sound and voice annotation, 

for exporting to Flash, and especially for adding on¬ 
screen text and comments. 


RealVNC 4.0 (viewer mode) Connect to: 131.120.147.110 password: noidus 

Right-click title bar for options. 

Useful options: view only, scale 


Event Logging Keyboard 
PJ. Engineering X-keys SE 

PIE002-13 Jog & Shuttle 
Editor 



X-keys Macro Works 


Used to code events in Noidus Observer, ffwd/rewind 
through videos in Observer (use ObserverControi.xk4 
file as a tempiate), and for video editing 


Sets up the event logging keyboard for use with 
different programs. 

The Noidus Observer basic control fiie, to use as 
tempiate for other configurations, has settings stored 
for using the jog/shuttie dial, but none of the keys: 
ObserverControl,xk4 


DivX codec and DivX player 



Adobe Premier Pro 1.5 

Note required updates: 

Main Concept MPEG encoding 1.1 
Windows XP KBa86716 


A highly compressed codec for AVI files that is 
popular for movies and video. 

Download free from: http://www.divx.com 

The codec must be installed on any computer that 
needs to play video files created with the Observer 
system. 


High-end professional video editing program. 
Expensive and difficult to use. it can do multiple 
picture-in-picture and other features not found in 
cheaper video editing programs, but is not easy to use 
for simple projects If you can take the time to learn it, 
it's the most powerful video editor available for 
Windows. 
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Participant station 


Room 221a Combination: 3,1,5 


Participant workstation 
computer 



Computer Name: IT004151 (on NPGS domain) 


IP address: 

SNM: 

gateway: 

DNS servers: 

MAC Address: 


131.120.147.110 (static) 
255 - 255 . 252.0 

131.120.144.1 
131.120.le.40 
131.120.13.41 
00-Q4-76-D7-72-EC 


Administrator account: admin 

password: HSILpa$$ 

Local user account: participant 

password: HSIL221a 


Windows 2000, 512MB RAM, 2Ghz Pentium 4. Two 18 GB 
drives 


VN C Serve r Allows the participant computer to be viewed or corTtrolled 

by the observer station. Setup VNC from the Admin account 
on the participant workstation. 

Pa n / tilt / zoom ca meras do not move cameras by hand - only by remote. Press 
^ 2 ^ <HOME> on remote if imoved by hand [see manual) 



Ceiling microphone Powered by mic amp 


Wireless microphone 
n 



Check 9V battery, have spares on hand Turn OFF after 
use. 


Mobile device ca mera tow battery will cause interference on video 
Check 9V battery, have spares on hand 


O 
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Office furniture 


Turn OFF after use. 

Can be used with cabte, harcFwired directly to an RCA video 
input (for higher quality), or through the MDC wireless 
receiver 


Check arrangement in relation to cameras and microphone 


O 
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System Hardware Components 



One-way mirrored glass 
Observer microphone 

Observer workstation monitor, keylxjard, mouse 

Camera joystick control 

Video mixer 

Event logging keyboard 


Program BA/V monitor 
Preview BAA/ monitor 
Wireless mic receiver 
Speaker and mic amps 



Observer workstation computer 
Printer 



Camera joystick control Video mixer 

Workstation mouse Event logging 
keyboard 



Mobile Device Camera (MDC) 
attached to a Ruggedized PDA 


miniDV camcorder mounted on mini 
tripod 


0 
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Wiring Diagram; VIDEO 
AWE Center Observation Laboratory 


Participant 

Room 




CAMERA I 
{front of roonn) 

RS232 m 




Observer 

Room 



JOYSTICK 

CONTROLLER 


-I 


S-VIDEO OUT 


RSZ32 0Lrr 

I 


RS232 IN 

CAMERA 2 

(side of room) 



VIDEO MIXER BACK PANEL 


o -OC^C 







1 S-VIOEOOUT 


MDC RECEIVER 



^ROA] 

U' 

Oh 


wlrelesG or wired connection 
(lo avoid RE interference) 


S-VH$ VCR 


■OQPO OOC 

oo66 o 5o6 



Aej^o_ Audi o S-V i1r^< 


I. - v 'J CANOPUS card 

i ■'I! ^^PICOLOcar 
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Participant 

Room 



Wiring Diagram: AUDiO 
AWE Center Observation Laboratory 


Observer 

Room 



CH t IN CH 2 IN 



WIRELESS MtC 
RECEIVER 
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Speaker amp 



Video mixer 


Note that you can use S-Video either/or RCA 
connectors on the four inputs, and must select it 
via the Route menu (MX-4 manual ch 7, pg 73) 


Micamp 


O 
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Screen Shots of Typical Configurations 

Useful desktop shortcuts to common apps (also found in Start Menu) 




Observer running full screen in 
background 

C a mta sia re co rder sta rt/sto p wi n d ow 
VNC to Participant workstation 

Preview of Participant room in Picolo 
C a me ra Vie we r w in do w 



Desktop icons 


Picolo Camera Viewer showing 
Picture-in-Picture (PIP) of two 
cameras 


Canopus recorder controller 


Canopus video output window (cannot 
be screen-captured) 

Canopus recorder controller (partially 
hidden) 
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Appendix D. Portable Observation Laboratory 
Embarkation List 

Computer Case 

(1) Laptop computer 

(1) Laptop computer A/C power adapter 

(1) USB mouse 

(1) 1GB USB thumb drive 

(2) blank CD-R media 

(1) DVD+RW burner 

(1) DVD+RW burner power adapter 

(1) DVD+RW burner USB 2.0 connecting cable 

(1) pad of writing paper 

(3) pens 
(3) pencils 
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Mobile Observer Case 1 (wireless) 

(1) Mobile Device Camera (MDC) 

(1) MDC receiver 

(1) MDC A/C power plug 

(1) MDC video cable (custom plug to RCA M) 

(1) MDC battery power supply 

(1) wireless mic transmitter 

(1) wireless mic lapel microphone 

(1) wireless mic receiver 

(1) wireless mic A/C power plug 

(1) wireless mic receiver cable (1/4" phono plug to dual RCA M) 

(1) portable miniDV VCR 
(1) portable miniDV VCR 
(1) portable miniDV VCR battery 

(1) portable miniDV VCR A/V input cable (3 RCA F to 1/8" plug) 
(4) spare 9V batteries for MDC and wireless mic 
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Mobile Observer Case 2 (camcorders) 


(2) miniDV-firewire camcorders 

(2) miniDV-firewire camcorder batteries 

(2) miniDV-firewire camcorder A/C power adapter 

(2) miniDV-firewire camcorder memory sticks for digital camera function 

(2) mini tripods 

(1) Large tripod ? 

(2) Voice recorders 
(2) External mics 

(1) spare battery for miniDV camcorders and miniDV VCR 
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Mobile Observer Case 3 (accessories and supplies) 

(2) surge protectors 

(1)6'-10' LAN cable 

(1) S-Video cable 

(1) 3-way RCA patch cable 

(1) Female-Female RCA connectors 

(1) full-size (6-pin) firewire cable 

(1) mini (4-pin) firewire cable 

(1)full size USB cable 

(1) mini USB cable 

(10) 9V batteries for MDC and wireless mic 

(20) AAA batteries for voice recorders 

(10) miniDV tapes 

(5) blank CD-R media 

(5) blank DVD+RW media 

(1) set headphones 

(1) S-Video to RCA adapters 

(2) BNCto RCA adapters 
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Appendix E: Useful signs and displays 


The following pages contain signs and displays that come in handy around the 
usability lab. 


The original files containing these signs can be found on the CD accompanying 
this user's guide. 
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Camera Controller Presets: Camera 1 (FRONT OF ROOM) 


1 - DOOR 

2 - DESK 

3-TABLE 

4-WHITE BOARD 

5 

6 


Camera Controller Presets: Camera 2 (SIDE VIEW) 


1 - DOOR 

2 - DESK 

3-TABLE 

4-WHITE BOARD 

5 

6 
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NOTICE; 

Audio and Video 
Recording Equipment 
may be IN USE 
in this lab 
at any time 


RECORDING 

IN 

PROGRESS 

quiet, please 


Participant computer workstation instructions 



O 

AWf . 


AWE Observation Laboratory Basic User Guide 
Appendix E: Useful Signs and Displays 


page 54 


142 















References 


Albers, M., & Kim, L. (2002). Information design for the small-screen interface: 

An overview of web design issues for personal digital assistants. 

Technical Communication, 49(1), 45. 

Booth, P. A. (1989). An introduction to human-computer interaction. Hove, UK; 
Hillsdale, USA: Erlbaum. 

Bordens, K. S., & Abbot, B. B. (2002). Research design and methods: A process 
approach (5th ed.). New York: McGraw-Hill Higher Education. 

Brewster, S., & Dunlop, M. (2004). Mobile human-computer interaction - 

MobileHCI 2004. Mobile human-computer interaction - MobileHCi 2004: 
6th international symposium, MobileHCi 2004, giasgow, UK, September 
13-16, 2004: Proceedings, xvii, 541. 

Card, S. K., Moran, T. P., & Newell, A. (1983). The psychoiogy of human- 
computer interaction. Hillsdale, N.J: L. Erlbaum Associates. 

Cooper, A. (1995). About face : The essentials of user interface design. Foster 
City, CA: Programmers Press. 

Cozby, P. C. (2004). Methods in behavioral research (8th ed.). New York: 
McGraw-Hill Higher Education. 

Fitts, P. (1954). The information capacity of the human motor system in 

controlling the aplitude of movement. Journal of Experimental Psychology, 
47(1), 381-391. 

Green, P. (2000). The Human Interface for ITS Display and Control Systems: 

Developing International Standards to Promote Safety and Usability (Utsu, 
Japan ed.). Ann Arbor, Michigan: University of Michigan Transportation 
Research Institute. Retrieved March 2005, from http:// 

Hackos, J. T., & Redish, J. (1998). User and task analysis for interface design. 
New York: Wiley. 

Harrison, M. D., & Thimbleby, H. (1990). Formal methods in human-computer 
interaction. Cambridge ; New York: Cambridge University Press. 

Head, A. J. (1999). Design wise: A guide for evaluating the interface design of 
information resources. Medford, NJ: Information Today. 

Hinckley, K. (2003). Input technologies and techniques. In J. A. Jacko, & A. 

Sears (Eds.), The human-computer interaction handbook: Fundamentals, 
evolving techniques, and emerging applications (First Edition ed.) (pp. 
1277). New Jersey: Lawrence Erlbaum Associates. 

International Organization for Standards (ISO) Technical Committee \SOfTC 159, 
Ergonomics, Subcommittee SC 4, Ergonomics of human-system 
interaction. (1990). /SO 9241-11 ergonomic requirements for o^ce work 


O 


AWE Observation Laboratory Basic User Guide 
References 


page 55 


143 




with visual display terminals (VDTs) - part 11: Guidance on usability No. 
9241-11). Geneva, Switzerland: International Organization for 
Standardization (ISO). 

Jacko, J. A., & Sears, A. (2003). The human-computer Interaction handbook: 
Fundamentals, evolving technologies, and emerging applications. 

Mahwah, N.J: Lawrence Erlbaum Associates. 

Johnson, J. (2000). GUI bloopers : Don'ts and do's for software developers and 
web designers. San Francisco: Morgan Kaufmann Publishers. 

Jordan, P. W. (1998). An Introduction to usability. London; Bristol, Pa: Taylor & 
Francis. 

Kim, L., & Albers, M. J. (2003). Presenting information on the small-screen 

interface: Effects of table formatting. IEEE Transactions on Professional 
Communication, 46(2), 94. 

Kim, L., & Albers, M. J. (2002). Web design issues when searching for 

information using handheld interfaces. Technical Communication, 49(3), 
314. 

Kirwan, B., & Ainsworth, L. K. (1992). A guide to task analysis. London; 
Washington, DC: Taylor & Francis. 

Lee, S., & Hong, S. H. (2004). Chording as a text entry method in mobile phones. 
In S. Brewster, & M. Dunlop (Eds.), Mobile human-computer Interaction - 
MoblleHCI 2004 (6th International Symposium ed.) (pp. 456). New York: 
Springer. 

Minasi, M. (1994). Secrets of effective GUI design. San Francisco: Sybex. 

Monk, A. (1984). Fundamentals of human-computer Interaction. London ; 

Orlando: Academic Press. 

National Research Council (U.S.). Panel on Human Factors in the Design of 

Tactical Display Systems for the Individual Soldier. (1997). Tactical display 
for soldiers : Human factors considerations. Washington, D.C: National 
Academy Press. 

Nielsen, J. (1993). Usability engineering (Updated ed.). San Francisco, Calif: 
Morgan Kaufmann Publishers, 1994. 

Noldus Information Technology. (2003). The observer reference manual version 
5. Wageningen, The Netherlands: Noldus Information Technology. 

Norman, D. A. (1998). The Invisible computer: Wty good products can faii the 
personal computer is so complex, and information appliances are the 
solution. Cambridge, Mass: MIT Press. 

Perlman, G. (2005). HCI bibliography: Human-computer interaction / user 
interface usability. Retrieved August 2005, 2005 from 
http://WWW, hcibib^org/ 


O 


AWE Observation Laboratory Basic User Guide 
References 


page 56 


144 





Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct 
effective tests. New York: Wiley. 

Schraagen, J. M., Chipman, S. F., & Shalin, V. L. (2000). Cognitive task analysis. 
Mahwah, N.J: L. Erlbaum Associates. 

Shneiderman, B. (2002). Leonardo's laptop: Human needs and the new 
computing technologies. Cambridge, Mass: MIT Press. 

Shneiderman, B. (1998). Designing the user Interface: Strategies for effective 
human-computer-Interactlon (3rd ed.). Reading, Mass: Addison Wesley 
Longman. 

Siewiorek, D. P., & Smailagic, A. (2003). User-centered interdisciplinary design 
of wearable computers. In J. A. Jacko, & A. Sears (Eds.), The human- 
computer Interaction handbook: Fundamentals, evolving techniques, and 
emerging applications (First Edition ed.) (pp. 635). New Jersey: Lawrence 
Erlbaum Associates. 

Simon, H. A., Egidi, M., & Marris, R. L. (1992). Economics, bounded rationality 
and the cognitive revolution. Brookfield, VT: E. Elgar Pub. Co. 

Spolsky, J. (2001). User Interface design for programmers. Berkeley, CA; New 
York, NY: Apress; Distributed to the book trade in the United States by 
Springer-Verlag New York. 


O 


AWE Observation Laboratory Basic User Guide 
References 




THIS PAGE INTENTIONALLY LEFT BLANK 


146 



APPENDIX C. APPLIED WARFIGHTER ERGONOMICS CENTER 

INFORMATIONAL MATERIALS 


1. NAVAL POSTGRADUATE SCHOOL PUBLIC AFFAIRS OFFICE 
ARTICLE 


New Applied Warfighter Ergonomics Center 
Measures Human-Machine Performance 

Friday, February 11,2005 Article by Javier Chagoya 


U .S. Maiine C oips C Koicli Takagi uses the 

N ddus Observation System in the new NFS 
Aprplied Warfi^ter Ergonomics Center (AWE) to 
gather empirical data about the interaction 
betweenhuman performance variables and a 
broad range of tactical equipment, including 
computer workstations, handheld defaces, and 
weapons. Participants are asked to perform 
specific tasks as they are observed through a one¬ 
way mirror, as well as by remotely-controlled 
distal video cameras^ microphones^ and computer 
m onitors that capture operator activity. 

Taka^ uses a special keypad to code participants' 
behavior. He's particularly interested in aspects of 
the machine-hum an interface considered positive 
or negative by users. F or instance, a participant 
might be aggravated by a poorly designed piece of 
equipment, or pleased by a specific function. 

Taka^'s vision is to empirically evaluate human- 
system performance variables related to 
equipment designed for warfighters, and make 
recommendations to desigiers leading to 
improved system performance andwarfi^ter 
safety. 

Phase I of the project was to acquire and then 
m aster the N ddus Observation platform, a 
collection of off-the- shelf components and 
proprietary software. The equipment has an eye¬ 
tracking system that records points-of-gase, 
identifies triggers of interest, and measures 
physical and mental workload variables^ stress^ 
dexterity, and reaction time. 

In the future;, Takagi plans to apply AWE's 
capabilities to cognitive task analyses using 
subject matter experts;, and to record focus group 
and after-action report sessions. The process will 
also evaluate man-machine interfaces and 
m easure the usabilily of hardware and software, 
including training effectiveness and retention 
AWE's Phase II will extend the system's Phase I 
capabilities to field environments. S om e 
equipment to achieve this is alrea(^ in place: a 
portable digtal camera, which attaches to a 
Tacticom, a hardened version of a Blackberry; and 



Caf]t Koichi Takagi sits at the observation console vihere he 
gathers measurement data on a participant's performance in 
the ne>i rocm. 


a portable computer system for performing on site 
analysis. 

After demonstrating AWE's capabilities to 
Professor^ Dave N etzer and Alex B or detsky F eb. 
S, the decision was made that a network ad^tion 
may soon follow. This wiU allow the system to 
transfer data seamlessly in the fields as in the 
laboratory. 

Phase III goals are to build an anthropometries 
and biomechanics laboratory to evaluate how 
military equipment impacts warfi^ters' ability to 
perform physical tasks. The lab wiU include a 
video-based motion capture system capable of 
statically and dynamically measuring size, 
strength, kinematics; range ofmotior; gait, and 
postural sway of warfi^ter subjects. 

Takagi is actively looking for participants; and 
can be reached at the HumanSystems Integration 
Laboratory ktakagifSfnps.edu . 
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APPLIED WARFIGHTER ERGONOMICS OBSERVATION 
LABORATORY DATA SHEET 



Applied Warfighter Ergonomics 
Observation Laboratory Dgta Sheet 
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jktkl 
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Portabte Observation Laboij'ato/y 
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APPENDIX D. POINTS OF CONTACT 


1. NAVAL POSTGRADUATE SCHOOL (NPS) 



Dr. Nita Miller 

Director, Human Systems Integration program 


Dr. Dave Netzer, Director, CDTEMS 

Tactical Network Topology (TNT) program director 


Marianna Verett 

Dr. Netzer's research assistant, 1ST student 


Dr. Alex Bordetsky (ITM) 
cell 831-521-9196 


Dr. Eugene Bourakov (ITM) 


Dr. Kevin Jones 

Mech Engr, TNT Aeronautics expert 


Ryan Hale 

Systems Engineering special contract research assistant 



Francisco Caceres 

Capt, USMC. ITM Student with RPDA knowledge and contacts 
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Maj Carl Oros, USMC 
1ST faculty 


Tom Hyunh 
John Osmundson 
Systems Engineering 

LT Lani Bostick 

NFS Supply Officer, Navy Region Southwest, x2236 
(past ballroom, 1st deck Hermann Hall by comptroller) 



Matt Simpson 

IstLt, USAF. HSI Student. 



Derek Read 

IstLt, USAF. HSI Student. 



Koichi Takagi 

Capt, USMC. OR / MBA Student 
Naval Postgraduate School, (OA Code 61) 

1411 Cunningham Rd, Rm GL-219 
Monterey, CA 93943-5221 

Chris White 

Navy Human Performance Center 
Jeff Roth a I 

Reference and Instruction Librarian for Operations Research 

(831)656-2344 

jrothal@nps.edu 
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2. RPDA MANUFACTURERS 

Points of contact to the specific RPDA manufacturers is available in a 
separate file, by contacting the Operations Research department at the Naval 
Postgraduate School. 


3. SPONSORS AND OTHER AGENCIES 

United States Army Special Operations Command 
(USASOC) 

Ft. Bragg, Fayetteville, NC 
Brian Kent (LtCol - PM at JSOC, handles money) 
brian.l.kent@us.army.mil (910) 243-0451, cell (910) 670-8976 
MAI (Mobile Access Infrastructure) 

Joint Tactical C4I 

Kyle D. Longcrier 
Telecommunications Specialist 
J-6 Technology, JSOC 
Ph: (910)243-0205 

Email: kyle.longcrier@us.army.mil 
longcrierk@soc.smil.mil 

Rapid Equipping Force (REF) 

Don Taylor 

donald.taylorl @belvoir.army.mil 
Project Leader 

US Army Rapid Equipping Force (Wexford Group Intnt'l) 

Civ: (703) 704-2247 

DSN: 654-2247 

Cell: (703) 200-7565 

or cell: (703)589-2150 

FAX: (703) 704-4224 

http://www.ref.army.mil/ 
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4. LABORATORY EQUIPMENT SUPPLIERS 

Noldus Information Technology Inc. 

751 Miller Drive, Suite-E-5 
Leesburg, VA 20175 

Phone: 703-771-0440/800-355-9541 

Fax: 703-771-0441 

Web: www.noldus.com 

John J. McGraw, sales engineer 
j.mcgraw@noldus.com 

Bart van Roekel, installation and technical Expert 
b.vanroekel@noldus.com 

Donna Smith, office manager 
d.smith@noldus.com 

Potter’s Electronics 

Michael Barber 
1612 Fremont Blvd 
Seaside, CA 
393-0277 

GPS Source Technical Sales and Support 

Richard Reap 
719-561-9520 
sales@gpssource.com 
www.gpssource.com 
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5. DEFENSE LANGUAGE INSTITUTE (DU) AT THE PRESIDIO OF 
MONTEREY (POM) 


Col Jeff Cairns 

Garrison Commander (SOF background) 
jeffrey.cairns@monterey.army.mil 

Marine Corps Detachment (MarDet) 

Executive officer (XO) 

Capt Raymond Servano 
ravmond.servano@monterev.armv.mil 

(831)242-5757 

MarDet S-4/Supply 

Source to temp-loan helmets, flak jackets, packs, 782 gear 
242-6854 

Deck Gunny 

POC for USMC casuals 

SSgt Boissoneau nathan.boissoneau@monterey.army.mil 
242-5873 

S-3 Operations 

SSgt Murray keevin.murray@monterey.army.mil 


229th Military Intelligence (Ml) Bn 
S-3 (Operations) 

SFC Swanson 
242-5295, 242-5208 

Source to temp-loan military equipment and coordinate DU 
Army "casuals" (students not currently in training). “DSSOPS” is 
in charge of all DU casuals. 

Chief Hernandez baltazar.hernandez@monterey.army.mil 
Person to send Casual Request Form to 

Williamson, John C. (SSG) 

John.Williamson@monterey.army.mil 
Person who handles casuals for 229MI Bn. 
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Bumann, Gregory (SSG) 

Gregory.bunnann@monterey.army. mil 
242-5796 

NCO in charge of B Co casuals. B Co may have better 
participants and availability for research. 

Drill Sgt Stroud 

Bldg 622 ("Bn Mailroom" etc.), 2nd floor 
From NFS: take Del Monte to Van Buren, left on Van Buren, 
right on Franklin St, enter DLI Franklin St. gate, take first left 
(Rifle Range Rd.), enter parking lot on left. 


Petty Officer McGuire david.mcguire@cnet.navy.mil 
PO Buchanan benjamin.buchanan@cnet.navy.mil 
242-7125 

In charge of Navy detachment casuals only. Might be easier to 
coordinate directly with them to request Navy-only casuals. 
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APPENDIX E. SURVEYS 

TECHNOLOGY EXPERIENCE SURVEY 


Participant ID # 


Technology Experience Survey 

Instructions: Please circle the response or enter an answer that most accurately describes your 
experience with the following technologies. If you are not sure about an answer, make your best 
guess. 


Cell Phone Experience 


1. Do you have a cell phone? (if NO, skip to question 7) 

[ YES ] [ NO ] 

2. Approximately how often do you talk on your 
cell phone? 

Please enter a number 

(hours per week) 

3. Approximately how often do you use text 
messaging on your cell phone? 

(hours per week) 

4. Approximately how often do you customize 
your cell phone (change ring tones, 
background pictures, etc.)? 

(hours per week) 

5. Approximately how often do you play games 
or use other programs on your cell phone? 

(hours per week) 

6. Approximately how many phone numbers are 
stored on your cell phone phonebook? 



Personal Digital Assistant (PDA) Experience 

PDAs include Palm Pilots, Pocket PCs, personal organizers, and similar devices. 


7. I use a PDA: (Please circle one) 

1 

2 

3 

4 

never 

used in the past, 

currently use, less 

currently use, 30 

used 

but not now 

than 30 min per week 

min per week or more 


8. My experience with a PDA stylus: (A "stylus" 

is a small pen used to write on a PDA screen.) 

1 

2 

3 

4 

never 

used in the past. 

used in the past, 

used in the past, and 

used 

But didn’t iike it 

and think they’re OK 

think they’re great 


Video Game Experience 


9. How many hours per week do you play video games? 

(hours per week) 

10. I play online video games. 

[ YES ] [ NO ] 

11. I play video games on a home computer. 

[ YES ] [ NO ] 

12. I play video games on a video game system 
(Playstation, X-Box, etc.). 

[ YES ] [ NO ] 

13. If you play video games, please list a few of your favorite video games. 
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Participant ID # 


Computer Experience (Please circle one) 


14. I have a computer at work that is primarily for my 
own use. 

[ YES ] [ NO ] 

15.1 have a computer at work that I share with others. 

[ YES ] [ NO ] 

16. I never or almost never use a computer at work. 

[ YES ] [ NO ] 

17. I use a computer in my living quarters. 

[ YES ] [ NO ] 

18. I use public computers (at a library, internet cafe, 
base facility, etc.) 

[ YES ] [ NO ] 

19. I often use several programs at the same time. 

[ YES ] [ NO ] 

20. If I had a problem while using a computer, I would: 

Number all that apply, In the order that you might perform them. 

a. Stop trying, and do something else. 

b. Read the manuals. 

c. Read the help files on the computer. 

d. Ask someone for help. 

e. Keep trying different ways to do it. 

21.1 use a computer... 

Please enter an approximate number 

(hours per week) 

22. I send and receive email messages... 

(hours per week) 

23. I send and receive email attachments... 

(hours per week) 

24. I use online chat or instant messaging... 

(hours per week) 

25.1 use the internet to find information... 

(hours per week) 

26. I shop or browse online... 

(hours per MONTH) 

27. I use online banking services... 

(hours per MONTH) 

28. I trust computers to store sensitive personal 
information. 

[ YES ] [ NO ] 
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Participant ID # 


Military Technology Experience 

(expert = qualified to teach the subject) 


l=novice-► 5= expert 


29. Military radio equipment. 

please circle one 

[N/A] [1] 

[2] 

[3] 

[4] 

[5] 

30. Military GPS (PLGRS). 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

31. Standard topographical map. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

32. Standard military compass. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

33. FBCB2 or Blue Force Tracker. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

34. Tacticomp (indicate when) 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

35. Commander's Digital Assistant or other handheld 
computer (please describe). 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 


Demographic Information 


36. Age (in years) 


37. Gender 

[MALE] [FEMALE] 

38. Pay grade (E-3, 0-2, etc.) 


39. Branch of Service (USA, USMC, etc.) 


40. Total years of enlisted service 


41. Total years of commissioned service 


42. Designator, MOS, or rating 

(e.g Sl/l/O, 0602 CommO, 11B Infantry, etc.) 


43. Level of education (please check one) 

_1. less than high school 

_2. high school graduate or GED 

_3. some college 

_4. college graduate 

5. graduate degree 

44. Current duty assignment or billet description 
(e.g. intelligence analyst, machine gunner, etc.) 


45. Special schools and qualifications (circle all that apply, and write in any others) 

[a. Ranger] [ b. Pathfinder ] [c. EIB] [ d. Air Assault ] 

[ e. NCO course ] [ f. SNCO course ] [ g. Computer Training ] [ h. HALO ] 

please describe 

Other courses: 
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HANDHELD DEVICE EVALUATION 


Participant ID # 


Handheld Device Evaluation 

Instructions: Please circle the response or write an answer that most accurately describes your 
experiences during today’s scenario. Please describe your experiences only during today’s scenario. 
Even if you evaluated a particular feature during a previous scenario, describe your new experience 
with the equipment today. Don't fill in the same answer as before, unless your opinion is exactly the 
same. 

Use additional space on the back of the form as needed. Your detailed additional comments are 
some of the most useful results we obtain in this study. Please be frank and honest - all responses 
will be kept completely anonymous. 


Scenario 



Communications and networking _ 

4. Description of the communications and networking features I used in this scenario. 


5. Number of text messages sent, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

6. Satisfaction with the text messages I sent: ease of input, speed and accuracy of entering text, 
knowing when a message was sent and received by the recipient. 

very some\A/hat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[11 [2] [3] [4] [5] [N/A] 

additional comments: 


7. Number of text messages received, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

8. Satisfaction with the text messages I received: ease of retrieving and reading, knowing when a 
message has been received, accuracy of messages. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[11 [2] [3] [4] [5] [N/A] 

additional comments: 
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Participant ID # |^ 

9. Satisfaction with the communications and networking functions in this scenario. 

very some\A/hat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


10. Satisfaction the reliability of the communications and networking functions in this scenario. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


User interface - software _ 

11. Satisfaction with the quality of still images displayed (how well could I interpret the images 
displayed). 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


12. Number of moving videos I viewed in this scenario, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

13. Satisfaction with the quality of moving video displayed (how well could I interpret the video 
displayed). 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


14. Number of audio files I listened to in this scenario, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

15. Satisfaction with the quality of audio files (how well could I interpret the audio presented). 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 
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16. Number of times I used the mapping functions during this scenario, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

17. Satisfaction with the mapping functionality (how well could I interpret the map, find myself 
and other places on the map, how it compared to a paper map). 

very some\A/hat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


18. Satisfaction with the situational awareness tools: how well I knew where myself, and friendly 
and enemy units actually were (not just where the computer said they were). Also, how well I 
knew when the information displayed was old or possibly incorrect. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


User interface - hardware _ 

19. Number of times I used the stylus to operate the device during this scenario, (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

20. Satisfaction with using the stylus during this scenario. Please comment on issues 
experienced. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


21. Number of times I used the buttons to operate the device during this scenario. . (best guess) 

_ a. # Attempted: _ b. # Successful: _ 

22. Satisfaction with using the buttons on the device during this scenario. Please comment on 
issues experienced. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 
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23. Satisfaction with one-handed operation of the device during this scenario. Please comment 
on issues experienced. 

very somevvhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


24. Satisfaction with removing the device from my gear or carrying position to use it during this 
scenario. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


25. Satisfaction with stowing the device on my gear when not needed during this scenario 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


26. Ability to use my personal weapon while operating the device. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


27. Ability to read the screen under various conditions (sun, glare, shadows, night, etc.) 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 
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Participant ID # 

Overall system _ 

28. Satisfaction with the reliability of the system to perform its functions in this scenario, without 
malfunctioning, crashing, freezing, incorrect displays, or other errors. List types of malfunctions 
experienced. 

very soine\A/hat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


29. Satisfaction with the physical ruggedness of the system to perform under the conditions of 
the scenario. List any breakdowns your device experienced. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


30. Overall impression of the system during this scenario. 

very somevwhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[1] [2] [3] [4] [5] [N/A] 

additional comments: 


31. Impression of how easy this system is to learn and use for me and people like me. 

very somewhat somewhat very 

dissatisfied dissatisfied neutral satisfied satisfied 

[11 [2] [3] [4] [5] [N/A] 

additional comments: 


Free comments 

32. Please take as much room as you require, on this sheet and on the back of the other sheets, 
to record any suggestions, concerns, and random thoughts that occurred to you during this scenario. 
Thank you for your full participation in this survey. 
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APPENDIX F. USABILITY TEST SCENARIO 


1. INSTRUCTIONS AND SETUP 

Instructions for participants 

This scenario is somewhat artificial, and some parts may not make sense 
tactically or realistically. Do your best to keep a tactical mindset and respond 
appropriately. As a patrol leader, you should write down any instructions you 
receive over the loudspeakers (not the video), because you may need to brief 
your patrol later. You should ignore any grid coordinates given in the video. 

You should take a knee, or otherwise react to the scenario as you might in a real 
situation. Please stay over the “X” on the floor. 

Your call sign is “Alpha-Six", or just "Six" 

You are talking to "Alpha Company HQ" on the radio and on the RPDA. 

If instructed to "tell your patrol" something, just read it out loud in a clear voice as 
if they were in front of you. 

Tell your patrol any messages you receive over the RPDA. 

Familiarize yourself with the following points on the map. When plotting, use the 
closest location on the map that makes sense to you. 

Plot routes directly from point to point - do not try to stay on the roads. 

Point Pinos 
Bird Rock 
Pescadero Point 
Pebble Beach Quarry 

AlphaSix Patrol Base (blue rectangle in Pebble Beach) 

Alpha Company HQ (blue rectangle at the Presidio) 

Naval Postgraduate School 

Monterey Peninsula Airport (use center of runways as point of reference) 
Monterey Commercial Pier (the easternmost pier) 

A "QRF" is a Quick Reaction Force. 
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Setup 

Plug S-VIDEO and AUDIO (headset) cables into laptop. 

Set up laptop for dual screen: 

Right click on desktop -> Properties -> Settings tab -> 
click monitor 2: Resolution 800x600, 

Extend my Windows desktop on this monitor. 

Run CDR 32: Combined Action in Iraq 
Screen mode: Dual-Screen 
Facilitator notes: Learn 
Language: English 

To play a video in the CDR, click on the line of text, not the icon. 
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2. CONVENTIONAL EQUIPMENT SEGMENT (Z) 

Tasks: 1c, 2c, 5c, 6c, 3c, 5c (2^^ time), 6c (2^^ time), 7c, 6c(3^^ time), 4c 


(Z - Conv. Equipment) 

Facilitator 

Participant 

Task 

Evaluated 

1. As a group, orient 
participants to the important 
points on the map (listed in 
Participant Instructions). 

2. As a group, play video of 
RPDA instruction to review 
functionality. 

3. Allow participants 10 
minutes to practice on RPDAs. 

4. Answer any questions about 
the lab environment or the 
RPDAs. 



5. Have participant dress into 
their gear and calibrate the 
RPDA. Take a pencil, notepad, 
protractor, and blank map. 

Write participant number on 
map and first page of notepad. 

6. Start recording on 
observer computer. 


(Scenario 

setup) 

7. Instruction: “State your 
participant number.” 

States participant number out 
loud. 


8. Instruction: "Break out your 
map and note taking gear and 
stand by for your patrol order. 
Disregard the coordinates given 
in the video." 

Play CDR note 13. (METT-T 
order) 

Play title screen (to take the 
METT-T order off of screen) 

Watch video 
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(Z - Conv. Equipment) 

Facilitator 

Participant 

Task 

Evaluated 

9. Orient participant to map: 
Instruction: "You are currently 
at the patrol base located at 
grid 939506." 

Listen, read map 


10. Instruction: “Send a radio 
check to Alpha Company HQ" 

Turn on radio. 

Send voice comm check. 

“Alpha Company HQ, Radio 

1c comm chk 

Radio: “Roger, over” 

check, over” 



“Roger, Alpha-Six, out” 


11. Instruction: “Plot a direct 
patrol route with two legs from 
the Patrol Base, 
to Cypress Point, 
ending at Pescadero Point. 

Give us a thumbs up when you 
are done.” 

Plots route on map 

2c plot rte 
take part¬ 
icipant off and 
on task as he 
reads info or 
receives 
instructions 

12. Instruction: Tell your patrol 
the six digit grid coordinates of 
each point.” 

Reads out loud three grids: 

939 506 

916 487 

938 465 

13. Instruction: “Tell your 
patrol the bearing and range of 
the first leg, and the total 
distance of the trip.” 

Reads out loud bearing and 
range and total distance. 

234°, 3100m 

Total: 6200m 

14. Instruction: “You take your 
patrol on the road’ 

Watch video. 


15. Play CDR note 17. (lED 
goes off) 

Watch video 


16. Instruction: “You currently 
are at Bird Rock. Get on the 
radio and tell Alpha Company 
you’ve been ambushed.” 

Play note 18. (sporadic fire) 

Radio: “Alpha company, 
we’ve been ambushed” 

5c send msg 

17. Radio: “Roger, dismount 
and engage the enemy. What is 
your current grid?” 

Reads paper map. 

Radio “1 am at grid 925 499 “ 

6c rev msg 

3c read grid 
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(Z - Conv. Equipment) 


Task 


Facilitator_Participant_Evaluated 


18. Play CDR note 20. (2 

Marines down) 

Instruction: “Request an 
URGENT MEDEVAC FOR 

TWO MARINES WITH 
GUNSHOT WOUNDS AND 
HEAVY BLEEDING” 

Radio: REQUEST AN 

URGENT MEDEVAC FOR 
TWO MARINES WITH 
GUNSHOT WOUNDS AND 
HEAVY BLEEDING 

5c send msg 

19. Radio: “Roger, medevac on 
the way.” 

Play CDR note 21. (Helo 
Medevac) 

Watch video. 


20. Instruction: “You are now 
at Pescadero Point.” 

Play CDR note 28 [Page 2]. 
(arrive at house) 

Watch video. 


21. Radio: "Alpha Six, what 
were the grid coordinates of 
your original patrol route?" 

Radio: Reads back grids from 
notebook. 

939 506 

916 487 

938 465 

6c rev msg 

7c read back 

22. Radio: “Be prepared to 
displace to rally point at the 
Pebble Beach Quarry grid 958 
491” 

Radio: “Roger” 

Copies down grid 

6c rev msg 

23. Instruction: “Tell your 
patrol the bearing and range to 
the new rally point”. 

Uses paper map. 

Reads out loud bearing and 
range: 

36°, 3300m 

4c brg rng 
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3. RPDA SEGMENT OF SCENARIO: RPDA-A 

Tasks: 1a, 5a, 6a, 3a, 4a, 5a (2^^ time), 6a (2^^ time), 7a, 4a (2^^ time) 


(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

1 . Instruction: “Stow your 
radio and map and deploy the 
RPDA at this time. Announce 
out loud to your patrol any 
instructions received on the 
RPDA. You currently are at the 
Naval Postgraduate School" 

Take out RPDA 


2. Instruction: “Send comm 
check” 

RPDA: “comm check” 

la comm, chk 

3. Play CDR note 30. (enter 
house) 

Instruction: “Send a message 
to Alpha Company HQ telling 
them you FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

Play sporadic gunfire 

Play note 31 (Police) 
repeatedly 

RPDA: ‘FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

5a send msg 

4. RPDA: “BRING SUSPECT 
TO HQ” 

Read RPDA. Say, “BRING 
SUSPECT TO HQ”. 

6a rev msg 

^ InotKi ■ “Dlr\f o rrM 

Plnt^ rniitp nn RPDA 

2a plot rte 
task omitted, 
not supported 
by RPDA 

vJ. 11 lo 11 LI w 11 w 11 > 1 IvJL Ca IvJULO 

with two legs from your 
location to the rally point and 
then back to A CO HQ at 
” task omitted 

riL-^Lo ivJULC vJi 1 r\r L/rA 

task omitted 

Instruction: “The new 
checkpoint is at the 
intersection of Hwy 68 and 

Route 1. The Alpha Co XO is 
at the commercial pier. 
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(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

6. Instruction: “Tell your 
patrol the 6-digit grid 
coordinates of each point” 
note that RPDA-A was not 
calibrated to the correct grid 
coordinates on a paper map. 

Reads out three loud grids 
from RPDA: 

A6. 

Checkpoint. 

A Co XO at commercial pier. 
(acceptable if the participant 
taps in the right vicinity and 
reads grid correctly) 

3a read grid 

7. Instruction: “Tell your 
patrol the bearing and range to 
the checkpoint, and the total 
distance of the trip 

Reads out loud bearing and 
range, and total distance 
(not supported by RPDA) 

121°, 566m 

4a brg rng 

8. Play CDR note 33. (angry 
mob) 

Instruction: “Send a message 
to Alpha Company telling them 
‘ANGRY MOB BLOCKING 
WITHDRAWAL’ 

RPDA: “ANGRY MOB 
BLOCKING WITHDRAWAL” 

5a send msg 

9. RPDA: “ROGER. QRF 
ENROUTE” 

Play CDR note 35. (QRF 
arrives) 

Say, “QRF Enroute”. 

Watch video 

6a rev msg 

10. Instruction: “The Alpha 
Company commander has 
arrived with the Quick 

Reaction Force. He asks you 
to read out loud the message 
that you previously sent on 
your RPDA about what you 
found at the house. 

Go through message log and 
read back message. 

7a read back 
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(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

11 .The A Co Commander 
instructs you to proceed 
directly to Monterey Airport, 
rather than go through the rally 
point. Tell your patrol the 
bearing and range from your 
location straight to the 

Monterey Airport." 

Reads bearing and range off of 
PDA: 

113°, 2000m 

4a brg rng 

12. Play note 36. 

Instruction: “The QRF 
subdues crowd and you return 
to base. The mission is a 
success. Stow your RPDA and 
your gear at this time. END OF 
MISSION." 

Stop video recording. 
Instruction: "Please return 
your pencil, protractor, map, 
and notepad to the table" 

Watch video. 

Return RPDA to table. 

Remove gear and exit room. 


13. Give participants the 
Technology Experience 

Survey to complete. 

Complete survey, dismissed. 
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4. 


RPDA SEGMENT OF SCENARIO: RPDA-B 


Tasks: 3a, 5a 


(RPDA-B) 

Facilitator 

Participant 

Task 

Evaluated 

1. Instruction: “Prepare the 
RPDA at this time.” 

Take out RPDA 


2. Instruction: “Find the grid 
coordinates of the commercial 
pier, the Monterey airport, and 
the Naval Postgraduate 

School.” 

Reads out loud grids from 
RPDA: 

994 510 

027 498 

008 508 

(grids may be considerably 
different on RPDA if the map 
image has not been aligned) 

3a read grid 

3. Instruction: “Go to the 

Start Menu and use the Text 
Chat application. Send a text 
message requesting an 
‘URGENT MEDEVAC FOR 

TWO MARINES WITH 
GUNSHOT WOUNDS AND 
HEAVY BLEEDING’ ” 

RPDA: REQUEST URGENT 
MEDEVAC FOR TWO 
MARINES WITH GUNSHOT 
WOUNDS AND HEAVY 
BLEEDING 

5a send msg 
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5. RPDA SEGMENT OF SCENARIO: RPDA-C 


(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

1 . Instruction: “Stow your 
radio and map and deploy 
the RPDA at this time. 
Announce out loud to your 
patrol any instructions 
received on the RPDA. You 
currently are at the Naval 
Postgraduate School" 

Take out RPDA 


2. Instruction: “Send comm 
check” 

RPDA: “comm check” 

la comm, chk 

3. Play CDR note 30. (enter 
house) 

Instruction: “Send a message 
to Alpha Company HQ telling 
them you FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

Play sporadic gunfire 

Play note 31 (Police) 
repeatedly 

RPDA: ‘FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

5a send msg 

4. RPDA: “BRING SUSPECT 
TO HQ” 

Read RPDA. Say, “BRING 
SUSPECT TO HQ”. 

6a rev msg 

^ InotKi “Dlr\f o rrM 

Plnt^ rniitp on RPDA 

2a plot rte 
task omitted, 
not supported 
by RPDA 

vJ. II loll LI wll wl 1 > 1 IvJL Ca IvJULO 

with two legs from your 
location to the rally point and 
then back to A CO HQ at 
” task omitted 

r ILJULC? kji 1 r\r 

task omitted 

Instruction: “The new 
checkpoint is at the 
intersection of Hwy 68 and 

Route 1. The Alpha Co XO is 
at the commercial pier. 
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(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

6. Instruction: “Tell your 
patrol the 6-digit grid 
coordinates of each point” 

Reads out loud grids from 

RPDA: 

938 465 

958 491 

974 513 

3a read grid 

7. Instruction: “Tell your 
patrol the bearing and range to 
the checkpoint, and the total 
distance of the trip 

Reads out loud bearing and 
range, and total distance 
(not supported by RPDA) 

4a brg rng 

8. Play CDR note 33. (angry 
mob) 

Instruction: “Send a message 
to Alpha Company telling them 
‘ANGRY MOB BLOCKING 
WITHDRAWAL’ 

RPDA: “ANGRY MOB 
BLOCKING WITHDRAWAL” 

5a send msg 

9. RPDA: “ROGER. QRF 
ENROUTE” 

Play CDR note 35. (QRF 
arrives) 

Say, “QRF Enroute”. 

Watch video 

6a rev msg 

10. Instruction: “The Alpha 
Company commander has 
arrived with the Quick 

Reaction Force. He asks you 
to read out loud the message 
that you previously sent on 
your RPDA about what you 
found at the house. 

Go through message log and 
read back message. 

7a read back 

11 .The A Co Commander 
instructs you to proceed 
directly to Monterey Airport, 
rather than go through the rally 
point. Tell your patrol the 
bearing and range from your 
location straight to the 

Monterey Airport." 

Reads bearing and range off of 
PDA: 

73°, 9400m 

4a brg rng 


178 




(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

12. Play note 36. 

Watch video. 


Instruction: “The QRF 

Return RPDA to table. 


subdues crowd and you return 
to base. The mission is a 
success. Stow your RPDA and 
your gear at this time. END OF 
MISSION." 

Stop video recording. 
Instruction: "Please return 
your pencil, protractor, map, 
and notepad to the table" 

Remove gear and exit room. 


13. Give participants the 
Technology Experience 

Survey to complete. 

Complete survey, dismissed. 
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6. RPDA SEGMENT OF SCENARIO: RPDA-D 


Tasks: 1a, 5a, 6a, 2a, 3a, 4a, 5a (2^^ time), 6a (2^^ time), 7a, 4a time) 


(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

1. Instruction: “Stow your 
radio and map and deploy 
the RPDA at this time. 
Announce out loud to your 
patrol any instructions 
received on the RPDA." 

Take out RPDA 


2. Instruction: “Send comm 
check” 

RPDA: “comm check” 

la comm, chk 

3. Play note 30. (enter house) 
Instruction: “Send a message 
to Alpha Company HQ telling 
them you FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

Play sporadic gunfire 

Play note 31 (Police) 
repeatedly 

RPDA: ‘FOUND SUSPECT 
WITH WIFE, SEVEN 
CHILDREN, TEN RELATIVES, 
ONE AK-47’ 

5a send msg 

4. RPDA: “BRING SUSPECT 
TO HQ” 

Read RPDA. Say, “BRING 
SUSPECT TO HQ”. 

6a rev msg 

5. Instruction: “Plot a route 
with two legs from your 
location to the rally point and 
then back to A CO HQ at 974 
513” 

Plots route on RPDA 

2a plot rte 

6. Instruction: “Tell your 
patrol the 6-digit grid 
coordinates of each point” 

Reads out loud grids from 

RPDA: 

938 465 

958 491 

974 513 

3a read grid 
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(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

7. Instruction: “Tell your 
patrol the bearing and range of 
the first leg, and the total 
distance of the trip.” 

Reads out loud bearing and 
range, and total distance: 

36°, 3300m 
total: 6400m 

4a brg rng 

8. Play note 33. (angry mob) 
Instruction: “Send a message 
to Alpha Company telling them 
‘ANGRY MOB BLOCKING 
WITHDRAWAL’ 

RPDA: “ANGRY MOB 
BLOCKING WITHDRAWAL” 

5a send msg 

9. RPDA: “ROGER. QRF 
ENROUTE” 

Play note 35. (QRF arrives) 

Say, “QRF Enroute”. 

Watch video 

6a rev msg 

10. Instruction: “The Alpha 
Company commander has 
arrived with the Quick 

Reaction Force. He asks you 
to read out loud the message 
about what you found at the 
house, that you previously sent 
on your RPDA. 

Go through message log and 
read back message. 

7a read back 

11 .The A Co Commander 
instructs you to proceed 
directly to Monterey Airport, 
rather than go through the rally 
point. Tell your patrol the 
bearing and range from your 
location straight to the 

Monterey Airport." 

Reads bearing and range off of 
PDA: 

73°, 9400m 

4a brg rng 
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(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

12. Play note 36. 

Watch video. 


Instruction: “The QRF 

Return RPDA to table. 


subdues crowd and you return 
to base. The mission is a 
success. Stow your RPDA and 
your gear at this time. END OF 
MISSION." 

Stop video recording. 
Instruction: "Please return 
your pencil, protractor, map, 
and notepad to the table" 

Remove gear and exit room. 


13. Give participants the 
Technology Experience 

Survey to complete. 

Complete survey, dismissed. 
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7. TACTICAL SCENARIO MAP 
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APPENDIX G. RPDA TRAINING 


1. RPDA-A LESSON PLAN BRIEF 


Advanced Soldier C2 Capabilities 
RPDA^s 



By: Mr. CyleFena 
Military Advisor 


1$AIJGUST2DD5 

8D1 mi 2814/ u irTD/omsrm^i 




ADVANCED SOLIDER C2 CAPABILITIES 


PURPOSE : The purpose of this training is to 
familiarize DLI Students with the basic 
functionality and characteristics of the RPDA-A 
software with an emphasis on situational 
understanding, voice communications, and text 
messaging through a multiplex radio RPDA- 
Architecture. 


1$ AUG 1672005 

SD im 1281V UUWCBDSr 0334 


YX 


ADVANCED SOLDIER C2 CAPABILITIES 



ADVANCED SOLDIER C2 CAPABILITIES 

AGENDA 



RPDA-A FAMILIARIZATION 

HISTORICAL BACKGROUND 



TASK; Become familiar with basic RPDA-A software 

TRAINING OBJECTIVES 



components. 

DESCRIPTION 




RPDA-AHITECTURE 



CONDITIONS: Given a classroom environment, one 

SOFTWARE FAMILIARIZATION 



primary instructor, all components of the RPDA-A and 

SYSTEM START UP 



RPDA-B systems with appropriate media, and training 

RPDA-A FUNCTIONS 




PRACTICAL APPLICATION 



STANDARDS: Become familiar with the software and all 

REVIEW/AAR 



components of the RPDA-A and TACICOMP system 




RPDA-Ahitecture. 

15AUGUST2JD5 Ij^JL 

8D1 mi 2814/ U UTD/[JBDSra334 T j 



15AUGU3T2m5 Ij Xj 

SDimi28i4/uu'nv[iBDS^a334 in 


ADVANCED SOLDIER C2 CAPABILITIES 
RPDA-A FAMILIARIZATION 


There are five basic components of the RPDA-A: 

Soldier PDA 
Radio (Short Range) 

GPS/ Battery Box Adapter 
Headset 

Push to Talk Button 


iSAUGusrmos 

8D1 mi 2814/ U Im]/Q8ns/D334 


YX 


ADVANCED SOLDIER C2 CAPABILTIES 


SOLDIER RPDA 

> Provides core capabilities 

- Mapping 

- Messaging 

- Voice over IP 

> Provides extended capabilities 

- PackBot Control 

- Robotic ATV Control 

- Weapons Control 

- Sensor Monitoring 



IS AUGU3T2m5 

SD im 12814/ UU WmOS/ 0334 


YX- 
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ADVANCED SOLDIER C2 CAPABILITIES 
DIGITAL RADIO 


• 802.11 b WiFi based radio 

- 2.4GHz, 11 channels (3 non-overlapping) 

• Network radio 

- Multi-Hop (>5) 

- Bridging/Routing 

• 2.4—0.6 mbps max throughput for 1-4 
hops 

• Internal battery with ~60 minutes 
reserve 

RF Range: ~400m LOS per hop 

Weight; 2 lbs 

Power: 3.2 W 

ISAUGUSraDDS 
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ADVANCED SOLDIER C2 CAPABILITIES 
GPS/ BATTERY BOX 

> Standard military battery 
with enclosed case 

- BA5590 family. 

> Cables with connectors are 
permanently attached to 
the battery box (‘flying 
leads’). 

> 12 Channel WAAS GPS. 

> Power conditioning. 

> Headphone jack for radio 
voice communications. 
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ADVANCED SOLDIER C2 CAPABILITIES 


Assembly and Start-Up 


YX 


ADVANCED SOLDIER C2 CAPABILITIES 
SYSTEM ASSEMBLY AND START UP 


TASK : Place the RPDA-Ainto Operation. 

CONDITIONS: Given a classroom environment, one 
primary and several assistant instructors, all assembled 
components of the RPDA-A system with appropriate 
media, and training aids. 

STANDARDS : Successfully assemble and initialize the 
RPDArAso that it provides Situational Understanding, 
VOIP Communications, Robotic Control, and Text 
Messaging. 


YX 


ADVANCED SOLDIER C2 CAPABILITIES 


ADVANCED SOLDIER C2 CAPABILITIES 


Components 


ASSEMBLY 



Showi with load-bearing vest 

15AUGU8TZ105 
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CAUTION: When installing cables do not crimp, force or pull cables. All 
cables are designed for a precision fit 
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ADVANCED SOLDIER C2 CAPABILITIES 
ASSEMBLY 


Note: Inspect all components for damage prior to assembly. 

1. Insert Battery into combination GPS/ Battery Box Housing. 

2. Secure butterfly connectors on the sides of the Battery Box Housing. 

3. Insert combination PTT/ Headset into cable labeled "Audio." 





ADVANCED SOLDIER C2 CAPABILITIES 


ADVANCED SOLDIER C2 CAPABILITIES 


ASSEMBLY 


ASSEMBLY 


1. InsertPrimary RPDA Cable into RPDA; place thin portion of cable 
through Cable Retainer and gently pull cable rearward. 

2. Slide cannon plug of cable forward, with the slotted portion up, until 
the plug is firmly seeded into the RPDA. 

3. Rotate the plug bezel in a clockwise motion until tightened. 

L ^ 



PDA Cable Assembly 


1. Connect quick Release radio plugs to GPS/ Battery Box Housing 
cable labeled "Radio." 



_J 

Quick disconnect for radio and headset 
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ADVANCED SOLDIER C2 CAPABILITIES 
ASSEMBLY 

1. Place two-way spring loaded radio power toggle in the" on" position. 
(Toggle should be pointed toward the antenna). 


E^ 



ADVANCED SOLDIER C2 CAPABILITIES 


TROUBLE SHOOTING 

PDA Screen Disappeais 

1. In the event the RPDA-A PDA 
screen disappears perform 
thefollovMng steps. 

2. Ensure the RPDA-A PDA cable is 
firmly seeded into the PDA. 

3. Check to see if Radio switch is in 
the “ON” position, antenna is 
connected securely, and has 
proper orientation. 

4. Ensure battery is secure and 
correctly inserted into GPS/ 
Batteiy Box Housing. 

5. Press “RESET” and re-enter 
correct organizational data, and 
re initialize RPDA-A. 

15 AUGUSTJODS 
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Loss of Communications 

1. Check all Audio cable connections 
to include PTT buttons to insure 
they are seeded and not depressed 
by equipment. 

2. Check volume settings on the PDA 
and in the “Symphone” 
application. 

3. Check “Symphone” application to 
insure correct “Talk Group.” 

4. Determine correct orientation of 
radio antenna, and insure radio 
switch is in the “ON” position. 

5. Go to “Tools” menu, select “Find 
Others” and list soldiers to 
determine if your PDA is reporting 
to the network. 
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ADVANCED SOLDIER C2 CAPABILITIES 
TROUBLESHOOTING 


Loss of ICONS or GPS 

1. Select ‘Tools” menu and select 
“GPS Status” to determine satellite 
connectivity. 

2. Check to insure the GPS node on 
the Battery Box Housing is not 
obstructed by equipment or 
overhead cover, and is oriented for 
best connectivity. 

3. Insure Radio cable is connected to 
the GPS Battery Box Housing. 

4. Turn off PDA, Turn off radio, 
remove battery from GPS/ Battery 
Box Housing. Check battery life. 

5. Re insert battery into GPS/ Battery 
Box Hoitsing and secure, turn 
radio powerswritch to “ON,” and 
initiate PDA by pushing soft 
“RESET.” 


6. Enter correct organizational data 
and inHialize RPDA.A; insure 
correct mapping image is posted. 

7. Wait for GPS to acquire, select 
‘Tools” menu, select “Find Others” 
and “List Soldiers” to insure your 
radio is reporting to the network. 




PDA Backlighting 

1. In the event the RPDA.A PDA 
screen is too bright or too dim 
perform the following steps: 

2. Select “Settings” from the main 
pull-dov/n menu in the upper left 
corner of the PDA screen. 

3. Select “System” at the bottom of 
the PDA screen, and choose the 
“Backlight” ICON. 

4. VUth your stylus, slide the 
indicator bar to the desired level 
of lighting and select “OK” in the 
upper right corner of the PDA. 


iSAUGUsraiDs 
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Loss of Sound 

1. Check all Audio cable connections to 
include PTT buttons to insure they 
are seeded and not depressed by 
equipment. 

2. Check the headset connectors for 
proper connection. 

3. Select “Speaker” ICON in the top of 
the PDAscreen and determine level 
of volume required. 

4. Push the large button in the lower 
right corner of the PDA and the 
“Symphone” application will appear. 

5. With the stylus, slide the VO III me 
control to the right to increase 
volume, to the left to decrease 



ADVANCED SOLDIER C2 CAPABILITIES 
TROUBLESHOOTING 


ADVANCED SOLDIER C2 CAPABILITIES 


START UP “RPDA-A” 




NOTE: Press soft "RESET” to clear RPDA-A memory prior to each use. 
This will automatically start the PDA. 

1. Press the "POWER" button firmly in the upper right comer of the soldier 
PDA for normal operations. 

2. The PDA should initiate and give you a "Symphone" screen. 

3. With your stylus touch the "X" in the upper right corner and you will see 
the "RPDA-A" start-up screen. 

4. With the stylus, touch the "Start" ICON in the upper left corner of the 
PDA. With your stylus, select "RPDA-A" to initiate the RPDA-A 

RPDAA Start Menu 


ADVANCED SOLDIER C2 CAPABILITIES 
START UP “RPDA-B” 


1. Power the RPDA-B by pressing the power indicator in the top right-hand 
corner of the RPDA 

2. Close the “GPS Network Settings” screen by single clicking the X on the upper 
right-hand corner of the screen. 

3. On the desktop, double click “ My Compirter.” 

4. Double click “Storage.” 

5. Double click “sql.” 

6. You will nowsee a message that says “Already Installed.” Chose “OK.” 

7. A navigation window will appear, chose “OK.” This will complete the installation of 
the SQL Server CE. 

8. Click the X in the upper right-hand side of the screen to close my computer. 

9. Double click “My Computer” once again and navigate to \Flash. 

10. Double click “Symphone,” then double click Symphone (a small icon vrith a 
lightening bolt through it. 

11. Once Symphone starts, the top will read “acquiring group”...wait until the screen 
reads “group acquired.” 


1SAUGUST2DD5 
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ADVANCED SOLDIER C2 CAPABILITIES 


START UP “RPDA-B” 

12. Minimize Symphone by single clicking 555-OOOx on the bottom of the screen. 
You will nowsee a small icon lightening bolt in the system tray area. 

13. Click the X in the upper right-hand side of the screen to close my computer. 

14. D ouble click “My Compirter” once again and navigate to \ Flash. 

15. Doubleclick “RPDA^” then double click RPD A-A vat h the blue icon to start the 
application. 


YX- 


ADVANCED SOLDIER C2 CAPABILITIES 
START UP 

1. The duty position and unit organization screen wiii appear. 

2. Enter your unit and individuai data to change position/task organization 
and tap "OK" using the styius. 

3. The RPDA-A appiication wiii begin ioading and give you the status. 

4. Once the appiication is ioaded, a map wiii appear with your current 
position indicated on the map in your squad coior with the appropriate 
weapons symboi. 

5. To verify your position, move in any direction to insure your iCON 
is in the correct iocation. 
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ADVANCED SOLDIER C2 CAPABILITIES 
MENU NAVIGATION 
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ADVANCED SOLDIER C2 CAPABILITIES 
MENU NAVIGATION 
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ADVANCED SOLDIER C2 CAPABILITIES 
ORGANIZATIONAL 

1. Display "List of Soldiers." This function allows you to see the task 
organization of your unit and who is operating on your network. 

2. Select “Tools," "ViewTree." and specific soldier; the map will 
"Auto Center" on that soldier. 

3. Select "Tools" and “Find Others" and the display will show who 
is currently operating on the network. 




YX 


ADVANCED SOLDIER C2 CAPABILITIES 


Mapping Data 




ADVANCED SOLDIER C2 CAPABILITIES 


MAPPING DATA 

1. GPS LEGEND 


Scale Bar (Meters)— 


UNIT 


COLOR 


Ten Di^ Grid - 

Topographic Map/ 


■-j . 

Imagery 


(Type Weapon) 

■Tl"7i 



SELF RED 

istSQD PURPLE 

ZndSQD LT.RED 

GREEN 

ROBOTS YELLOW 

BAD GPS GRAY 

REPORTING 

2. Scale Bar at the top is graduated in meters. 

3. Ten digit MGRS coordinate is at the top of the screen. 

4. Zoom "In" and "Out" at the bottom of the screen. 

5. North is always the top of the PDA so orient yourself accordingly. 

6. The GPS updates every 5 seconds. 


YX 
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ADVANCED SOLDIER C2 CAPABILITIES 
ICON FUNCTIONS AND OPTIONS 



RPD/i^APDA Screen vwth Situational 
Understanding 

IS AUGUST ZDS 
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ADVANCED SOLDIER C2 CAPABILITIES 
ICON FUNCTIONS AND OPTIONS 



IP Address: 

123456789 

Latitude: 33 70041 

Longitude: 

112.091786 

Status: 

Online One PS 


isAUGUsrars 
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With stylus, touch “Send Msg” (Message) and 
the text messaging program will appear 
allowing the user to send a text message 
without having to navigate back through the 
main menu. 

With the stylus touch “P2P Call” and the RPDA-fl 
will allow the user to initiate a point to point 
call. To end the call simply touch “P2P Call” 
again. 

With the stylus touch “Show Details” and the 
RPDA-A will show name, rank, duty position, 
location, IP address, and status of GPS, and 
digital radio. 




ADVANCED SOLDIER C2 CAPABILITIES 


MAPPING DATA 

7. Select "Map" and "GPS Status;" this screen will give you time 
since last GPS update, number of satellites currently tracking 
your position, ground speed, elevation, ZULU time, position in 
MGRS, and position in LAT/ LOG. 

MAP CALIBRATION 

1. Find a well defined location on the map and preposition yourself there. 

2. Refine your position by using the "Zoom" bar at the bottom of the 
Screen. 

3. Select "Map" and select "Start Calibration" and you will be prompted 
with instructions. 

4. Follow the instructions step-by-step. 

jAUGUsraES TT'^ 
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ADVANCED SOLDIER C2 CAPABILITIES 


MAP CALIBRATION 

5. With your stylus, touch the designated location on the map. 

NOTE: Be very precise in designating your new location. 

6. The RPDA-A will prompt you to restart the system. 

7. Select "Menu" at the bottom of the screen, and select "Exit." 

8. Select the "Start" ICON at the top of the PDA, select "RPDA-A" to restart 
the system. Once restarted follow the start procedures and continue 
operations. 



ADVANCED SOLDIER C2 CAPABILITIES 


Text Messaging 
and 

Ancillary Features 
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ADVANCED SOLDIER C2 CAPABILITIES 
TEXT MESSAGING 

Pfe-PofmaBed Messaging 
Twt Messagpng 
irscoming/ Outgoing Mail 


1. Selecf'Menu" and “Outgoir^g." 

2. Select "Keyboard" and begin typing message 
Un-select keyboard when typing is completed. 

3. SelectIbeTO'pulldown menu and you will 
be prompted with active addresses. 

4. Select address of the intended recipient and 
seleOt “SentT at ttie bottom of your screen 

tnconJfrtd 

1. Selec* "Menu" and "Incoming." 

2. Select tbe specific message and read. 

3. To reply select "Reply" or "Forward." 



L 
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ADVANCED SOLDIER C2 CAPABILITIES 

POCKET TERP FEATURES 

* Select Menu end Internet Euptorer. 

' Select language (Pashtu or Aiabrc). 

’ Wiih the "Stylus" tap Village Greetings (a list of standard phrases 
should appear). 

' Select a specific phrase 

* Tap "PLAY" with the Stylus to listen to the language translation. 

' Tap blue "Back Arrow at boBom of screen to return to previous 
list). 


' Tap Home ICON to go to Soldier Information. 

umaufifyi 
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ADVANCED SOLDIER C2 CAPABILITIES 
SUNIMARY 

HISTORICAL BACKGROUND 
TRAINING OBJECTIVES 
DESCRIPTION 
RPDA-AHtTECTURE 
SOFTWARE FAMILIARIZATION 
SYSTEM START UP 
RPDA-A FUNCTIONS 
PRACTICAL APPLICATION 
REVIEW/AAR 

siln» 17'^ 

sj*ruyraf[iii&[HU p ^ 



QUESTIONS? 
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2. RPDA-B STARTUP INSTRUCTIONS 

1. Press power button. 

2. "GPS Network Settings" - Tap [OK], 

3. Tap [Stop GPS]. 

4. (Double tap to open/activate icons, like a double-click in Windows): 
Calibrate > [Calibrate] > ignore arrow on screen, line up stylus tip with 
target. 

5. Repeat Step 4 until satisfied, close Calibrate app. 

6. My Computer > Flash (folder) > ARC (folder) > ARC (application). 

7. Click [Ok] on the three (or more) error message boxes. 

8. Use ARC program as with RPDA-A: set Manual GPS. 

9. NOTE: Using Menu > Mail to chat will crash the device. 

10. For Chat: Start Menu > Text Chat. 

11 .Tap in text area, call up keyboard (icon in lower right). Tap send. 

12. Close Chat window when done. 

13. To shut down or restart RPDA, use Start Menu > Suspend. 
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3. RPDA-C LESSON PLAN BRIEF 

Note: additional teaching notes can be found in the “presenter’s notes” in 
the original Powerpoint files. 


o 

AWE 


Basic 
RPDA 
Functions 
(RPDA C) 


o 


Lesson Plan 


• Overview of device C 

• Messaging 

- Send message 

- Receive message 

- Recall old messages 


O 

AWE 


Instructor 


• Mapping 

- Locate own position 

- Determine bearing and range to point 

- Orient on map features 


Overview 


o 


Messaging 


' Only a small portiori of full features will be covered 

• Power On, Off (leave ON) 

' Screen, stylus 

' Tap, double-tap (stylus) 

' Short push, long push (buttons) 

• Virtual Keyboard 

• Main applications: ArcPad and Text Chat 

- ArcPad: Short push on “4” button to launch 

- Text Chat: Long push on “2” button to launch 

• Look for “X” button in upper right corner to close 
windows 

• It will not be necessary to use other applications on the 
RPDA 


• Hold dovwn ‘'2' button until Text Chat window 
appears 


- If "2" button Is pushed too briefly, a different 
application will open. Close this application using the 


“X" 


Launch virtual keyboard with a firm tap on the 
stylus icon (2^^ from right in bottom taskbar) 

Tap in the white area of the Text Chat window to 
see the blinking cursor and then use virtual 
keyboard to enter message by tapping firmly on 
letters 


• Tap Send 
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o 

awe 


Messaging 


O 


Mapping 


‘ Scroll through gray area in Text Chat window to 
read previous messages (both sent and 
received) 

• Can scroll and move windows 

• DO NOT hit the “X” box on the Text Chat 
window when switching to another application; 
leave it open. Doing otherwise will erase your 
message log and undermine your ability to 
receive messages 


- Short Push on the button will start 
Arc Pad application 
• Relevant Menus are: 

- 2^^ Row, 4^ from left: Measure, Advanced 
Select (bearing, range, and grid coordinates) 
-2^^ Row, on left: Zoom In, Zoom Out, Pan 


• To close the virtual keyboard, firmly tap on the 
keyboard icon (2'^^ from right in bottom taskbar) 


O 


Grid Coordinates 


o 


Bearing and Range 


* Advanced Select (6^^ in appropriate pull 
down menu) 

* Tap on the desired point. 

• In the resulting Feature Information box, 
tap the “+” next to XY Coordinates. 

• Scroll down to the bottom for MGRS 
coordinates. 


■ Select Measure from the appropriate pull 
down menu (2"^). 

■ Tap on each of the two points using the 
presented ruler icon. This should create a 
solid red line between the two points. 

• Tap on Measure again (in the taskbar) to 
bring up the Measurement Information 
window and find the bearing and range 
(From -> To and Distance) of the trip. 
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o 

awe 


Zooming and Panning 


O 

tm.. 


Practical Application - 
Messaging 


• Be sure to select the" Pan” feature before 
trying to scroll around or tap on map 


- Start Text Chat, open Arcpad, return to 
Text Chat 


• To zoom in or out, select the appropriate 
feature and tap on the desired center of 
the zoomed image 


• Send a comm check 

* Send three and receive three text 
messages 


* Be patient when using these applications 


* Recall the first message you sent (using 
the message log). 


O 


Practical Application - 
Mapping 


* Open/Return to ArcPad 

* Find the grid coordinate of a specific point. 

* Use zoom and pan to find various points. 


• Find the bearing and range from one point 
to another. 
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RPDA-D LESSON PLAN BRIEF 


Lesson Plan 

Overview 

- Overview of device D 

• Messaging 

- Send message 

- Receive message 

- RecaH old messages 

• Mapping 

- Locate own position 

- Determine bearing and range to point 

- Orient on map features 

• Only a small portk>n of full features will be covered 

- Power On. Off (leave ON) 

- Screen, styijs 

- Tap, tap and hold, double-tap 

- Virtual Keyboard 

- Main application: C2CE 

- Slarl-> CTCfc (il necessary) 

• Look for 'OK" button in upper right corner to dose 
windows 

‘ DO NOT attempt to use other applicatbns on the RPDA 

Messaging 

yj Message Log 

- Start in C2CE Main Window 

• Tap handset icon ( 4 ^^ from left) 

• Tap radio with green check for “Comm Check” 

• Tap in Composer area (so that cursor blinks 
there) 

• Use virtual keyboard to enter message 

• Tap Send All (envelope All) icon 

• Scroll through Communications Log area to read 
previous messages 

• Tap OK (in upper-right corner) to close 

• Start in C2CE Main Window 

• Tap Yellow Pad icon (5*^ from left) 

• You can display, all messages, only Transmitted 
(TX), or only Received (RX). 

• List shows time sent or received. 

• Select message and tap View. 

• Tap Tx to transmit selected message again. 

• Can scroll and adjust column widths. 
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m Mapping 

Routes 

- Start in C2CE main screen 

- Icons are {left to right): 

- Pointer: display grid of the location you tap 

- Range Bearing: ffom/to two points on map 

- Drav/ (ignore) 

- Route: create waypoints in a route 

- Pan: tap and drag to move tJie map around 
-Zoom In (+), Out(-), Zoom Box 

‘ Tap Route icon (F?5. 

‘ Tap points along route. 

‘ Tap Pointer (arrow) when done. 

‘ Tap Ffts menu {at bottom of screen), detect Rts Mgr 

’ Select the Q JickRoute you just created (highest nui^iber) 

- Tap Edit 

• Use Scroll bar and/or adjust column v/rdth to see the 
position of each waypoint on the route 

- “Course" column shows the bearing from this waypoint to 
the next, "Leg Length" shows the distance. 

‘ Totar cofumn shows the total distance up to that 
waypoint. 

fiV Practical Application - 

Messaging 

nj Practical Application - 
MaoDina 

• Start C2CE, close C2CE, start it again 

• Send a comm check. 

• Send three and receive three text 
messages. 

• Recall the first message you sent (using 
the message log), and record the time it 
was sent. Send it again. 

• Find the grid coordinate of a specific point 

• Use zoom and pan to find various points. 

• Find the bearing and range from one point 
to another. 

• Create a route with at least three legs. 

• Identify the bearing and distance of the 
first leg. 

• Find the total distance of your route. 
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5. RPDA AND BASIC LAND NAVIGATION PRACTICAL APPLICATION 

Perform each task using both conventional means (i.e. paper map and protractor) 
and using the RPDA. Write down notes on how to do each task, and bring this 
sheet with you to the testing. 

□ Start RPDA main application, close it, start it again. 

□ Send communications check 
With a partner: 

□ Send three and receive three text messages. 

n Recall the first message you sent, and record the time it was sent: 


□ Send the first message you sent again. 

n Find the grid coordinate of a specific point, use zoom and pan to find various 
points ( choose two): __ 


Carmel River State Beach 


Asilomar State Beach 


Monterey Airport 


Salinas Airport 


Moss Landing 



n Find the bearing and range from the two points you found above. 
Bearing:_ Distance:_ 


n Create a route with at least three legs. 

□ Identify the bearing and distance of the first leg. 

Bearing:_ Distance:_ 

n Find the total distance of your route. 

Total Distance: 
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APPENDIX H. INFORMATION FOR PARTICIPANTS 


1. RESEARCH BRIEF 


O 

AWE 

Applied Warfighter Ergonomics 


Human Factors 
Study of 
Ruggedized 
Personal Digital 
Assistants 
(RPDAs) 


o 

AWE 


Introduction 


• What this study is (and is not) 

• What we will do today 

• What we will do later this week 

• Admin details 


Capt Takagi, USMC 
Lt Read, USAF 
Lt Simpson, USAF 


o 

awe 


What is this study for? 


• Testing four Ruggedized Personal Digital 
Assistants (RPDAs) in use or being considered 
by US Army Special Operations Command 
(USASOC) and USMC units in Iraq and 
Afghanistan. 

• We are primarily testing the “user-friendliness” of 
the software (not the hardware, networking, etc.) 

• We are not the developers, and are not 
advocates of any particular system. 

• There are significant limits to what we are 
studying, but we will report all comments on to 
the program managers. 


O 

AWE 


What is this study for? 


• Why so few devices? 

- Most are committed to operational forces. 

• Why does_seem broken? 

- These are prototype units. 

• Will I ever use one of these? 

- It’s quite possible. For example, there are 
variations specialized for translation. 
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Today 


gal Later this week... 

• 2-3 hour training session on a single 


• Each participant will come to NPS to 

Ruggedized Personal Digital Assistant 


operate the RPDA for a single 1-2 hour 

(RPDA). 


Individual session, which will be 

• Review of similar tasks in the “traditional” 


videotaped. 

method (radios, paper maps). 


• Will schedule small groups at one time 

• Schedule testing sessions for later this 


(prefer 2-6 individuals). 

week. 


• Lab is available 0700-0900 and 1000-1800 



Wed and Thu, and 0700-1900 Friday. 


o 

AWE 


Admin 


O 

AWE 


Privacy and Safety 


• Any questions about your packet? 

- Keep these: 

• Participant Information Form 

• Card with your Participant ID# 

- Turn in when you come to NPS (not today) 

• Three consent forms (signed) 

• Participant ID form ("For Official Use Only") 

• Technology Experience Survey 


• Your name will not be used anywhere In the 
study or the final report. 

• Please speak freely so that your feedback can 
be used to improve the devices. 

• If you choose not to participate, simply let your 
NCOS know. I will not know who did or did not 
participate in the study. 

• During this study, there should be no more risk 
to you than you are exposed to in your current 
daily routine. Please Inform any researcher if 
you see any unsafe conditions. 
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2. LETTER TO REQUEST PARTICIPANTS 



Applied Warfighter Ergonomics / Human Systems Integration 

Department of Operations Research Naval Postgraduate School 

Glasgow Hall rm 221 

lab phone: (831)656-3134 


I am a graduate student conducting Human Factors research at the Naval Postgraduate School. 
We are seeking participants in equipment testing. Specifically, we are testing ruggedized 
personal digital assistants (RPDAs - basically handheld networked tactical computers) for the 
Special Operations Command (SOCOM). Some of the devices are in current use by SOCOM in 
Iraq and Afghanistan. 

We are seeking DLI students, available between now and mid-August, to participate in this 
important, real-world testing. They would definitely benefit from exposure to state-of-the-art 
equipment that is already filtering down from SOCOM to regular units, and by participating in the 
tactical scenarios (based on current operations in Iraq and elsewhere) that we are developing. 

Our testing would involve: 

1. Approximately two hours of training on the equipment in a group session. 

2. Approximately one hour per individual to observe the students, one at a time, using the 
equipment in simulated tactical scenarios. 

Ideally, participants would be military personnel with very basic training in tactical procedures, 
such as wearing body armor, reading a map, and talking on a radio. The training any Marine or 
Soldier receives in boot camp would be sufficient. Personnel from other services, and even 
civilians (including those who have never been exposed to any tactical training) would also be 
welcome. Participants should not be "specially selected" on the basis of being more or less 
proficient than others - the equipment, not the participants, is being tested. The participants 
should be representative of all possible users, skilled and less skilled. 

To be a rigorous test, we would like to have twenty to forty individuals, who would be available as 
described above. Each individual participant would only have to be available for the initial half-day 
training in small groups, and then for a few hours sometime in the next week or so. Since each 
participant must be observed and recorded individually, it will take several days to work through 
all the participants, one at a time. 

We are very flexible about when and how this could occur, between now and maybe mid-August. 
We could also do this as a series of smaller groups (as few as four to ten individuals, in several 
blocks over the period), if that would be easier to schedule. 

The attachment describes our laboratory equipment. I would be happy to demonstrate our 
laboratory, or brief this experiment, to anyone who may be interested in more information. I can 
best be contacted via email: ktakaqi@.nps.edu . 

Very Respectfully, 

Koichi Takagi 
Capt USMC 

email: ktakaqi@nps.edu 
home: (831)372-5912 


205 





THIS PAGE INTENTIONALLY LEFT BLANK 


206 



3. 


PARTICIPANT INFORMATION FORM 



To all research study participants: 

You are invited to participate in a research study at the Naval Postgraduate School. Your 
participation will help improve equipment currently in use by Special Forces in Iraq and 
Afghanistan. Please read the description of the research on the following three pages, and initial 
and sign where requested. 

This is not a test of your skills or abilities. I want to emphasize that your participation or 
non-participation in this study will have no effect on your current or future military status. The 
equipment, not the individual participants, is being evaluated. There is no question of you, the 
participant, doing "well," or "poorly," and neither your participation nor your individual results will 
be entered into your military record, or released to your current command. 

Please do not discuss any part of the study with other students, or potential participants, 
until the study has concluded (around September, 2005). We want all participants to be equally 
prepared during the study, and no-one to be "coached" or otherwise prepped beforehand. 

During the study, you will receive approximately two to four hours of training (as a group) 
on a piece of military equipment. Please do your best to learn the gear, and ask as many 
questions as you like of the instructors or researchers. Within a few days after the training, you 
will be brought to the Naval Postgraduate School for an individual session of less than two hours. 
You will operate the equipment you were trained on while wearing body armor, helmet, etc., and 
watching a tactical scenario on a large plasma-screen TV. Many portions of the study will be 
videotaped, and so there is a release form that we ask you to sign so that we may use the 
recordings. These standard consent forms are always required in any kind of research on people. 
I do not believe there is the slightest risk to your health or to your privacy. 

Please fill out the enclosed survey. If you have any questions about the survey, we will 
answer them during the group training session. 

Please don't hesitate to ask us any questions you may have at any time during the study. 
Thank you for your participation. 

Sincerely, 


Capt Koichi Takagi 
United States Marine Corps 
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4. 


CONSENT FORMS 


PRIVACY ACT STATMENT 

I^AVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
PRIVA CYACT STA TEMENT 

1. Puipose: Data from this research. Evaluating the Human Factors of Military Handheld 
Computing Devices^ will be used to develop tests, tactics, procedures, and equipment to 
improve the human-machine interfaces of military handheld computing devices. 

2. Use: Research data from this study will be used for statistical analysis by the Departments 
of the Navy and Defense, and other U.S. Government agencies, provided this use is 
compatible with the purpose for which the information was collected. Use of the 
information may be granted to legitimate non-govemmeni agencies or individuals by the 
Naval Postgraduate School in accordance with the provisions of the Freedom of Information 
Acl 

3. Disc lo sure/C on fidentiality: 

a, I have been assured that my privacy will be safeguarded. I will be assigned a control or 
code number which thereafter will be the only identifying entry on any of the research 
records. The Principal Investigator will maintain the cross-reference between name and 
control number. It will be decoded only when beneficial to me or if some 
circumstances, which is not apparent at this time, would make it clear that decoding 
would enhance the va.]ue of the research data. In all cases, the provisions of the Privacy 
Act Statement will be honored. 

bn I understand that a record of the information contained in this Consent Statement or 
derived from the experiment described herein will be retained permanently at the Naval 
Postgraduate School or by higher authority. I voluntarily agree to its disclosure to 
agencies or individuals indicated in paragraph 3 and I have been informed that failure to 
agree to such disclosure may negate the purpose for which the experiment was 
conducted. 

c. 1 also understand that disclosure of the requested information, including my Social 
Security Number, is voluntary. 


Name, Rank (please print) 

Signature of Voluntary Participant Date 
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PARTICIPANT CONSENT FORM 


1 iiiiiiai:_Introduction. You are invited to panicipaie in a study of military handheld computing devices. 

infontiaiion gathered from you and other participanlSj we hope to discover the human factors considerations 
pertaining to these dev ices. Please read and sign this form indicating that you agree to be in the study. Feel free to 
ask any questions you may have before signing. 

2. iniiai:_Background information. The Naval Postgraduate School Human System Integration Laboratory 

and Applied Warfighter Ergonomics Center are conducting this study. 

3. Initial; _Procedures. If you agree to participate in this study, the researcher will explain the tasks m detail. 

There will be several sessions consisting of written questionnaires, equipment training and operation, and 
participation m multimedia tactical scenarios. 

4. initial:^_Risks and Benefits, This research involves no risks or discomforts other than: answering questions 

(on paper and in person) about your experiences w ith various equipment during field exercises and/'or laboratory' 
trials. You may be asked to wear standard military individual equipment. Audio and video recordings of your 
participation will be made if you consent (item 9 below). Your participation w'ill significantly contribute to 
improving the effectiveness of equipment currently being fielded in the U S. Dept, of Defense and to current 
research in human-machine interaction. 

5. Initial:_Compensation. No tangible rew'ard wdll be given. A copy of the results wifi he available lo you at the 

conclusion of the experiment. 

6. initial;_Confidentiality, The records of this study will be kept confidential. Special care will be taken to 

preses^'e anonymity in all data collection. Written documentation will be identified by participant number only. 
No information w-hich could identify you as a participant w'ill be made accessible to your military unit, your 
supervisors, or to the public. 

7. Initial:_Voluntary Nature of the Study. If you agree to participate, you are free lo withdraw from the study 

at any time without prejudice. You will be provided a copy of this form for your records. 

S. Initial:_Points of Contact. If you have any further questions or comments after the completion of the study, 

you may contact the principal researcher, Nita Lewis Miller, PhD, at (glil) 656-22S1, DSN: 756-2281, 
nlmi 11 err^rfiips.cdn . or Capt Koichi Takagi, USMC, at (831) 656-3134, ktakattir^.'nns.edu . 

9. Initial:_Consent to Video and Audio Recording. J agree that audio and video recordings of my panicipation 

in thi.s research may be made. 1 agree that the recordings may be: 

Initial: _Studied by members of the research team, 

Initial:, _ShowTi to Other researchers, research participants, and students, and 

Initial:_Used in future training. 

10. Initial:_Slatement of Consent, I have read the above information. Any questions I had concerning this 

research have been answered to my satisfaction, I agree to participate in this study. 


Sigr ature of Researcher 




Signature of Voluntary Partietpant 
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MINIMAL RISK CONSENT STATEMENT 

NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 

Participant; VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN: E^-alu3ting the Human Factors 
of Military Handheld Computinii Devices 


1. ! have read, understand and been provided ^‘Information for Participants" and ’’Participant Consent 
Form" that provides the details of the below acknowledgments. 

2. I understand that this project involves research. An explanation of the purposes of the research, a 
description of procedures to be used, identification of experimental procedures, and the extended 
duration of my participation have been provided to me, 

3. 1 understand that this project involves only minimal risk. I have been informed of any reasonably 
foreseeable risks or discomforts to me, 

4 I have been informed of any benefits to me or to others that may reasonably be expected from the 
research. 

5. I have signed a statement describing the extent to which confidentiality of records identifying me will 
be maintained. 

6. 1 have been informed of any compensation and/or medical treatments available if injury occurs and is 
so, w'hat they consist of, or where further information may be obtained. 

7. I understand that my participation in this project is voluntary, refusal to participate will involve no 
penalty or loss of benefits to w'hich I am otherwise entitled- 1 also understand that I may discontinue 
participation at any time without penally or loss of benefits to which I am othervv'ise entitled. 

8. I understand that the individual to contact should I need answ^ers to pertinent questions about the 
research is Professor Nita L. Miller, Principal Investigator, and about my rights as a research 
participant or concerning a research related injury is Prof. Jim Eagle, Operations Research Dept. 
Chairman or LTC Eric Morgan, MC, USA, Commanding Officer of the Presidio of Monterey Health 
Clinic. A full and responsive discussion of the elements of this project and my consent has taken 
place. NPS Medical Monitor: LTC Eric Morgan, MC, USA, eric.mortianrnw.amedd.armv .mi 1. 
(831) 242-7550, doctor on-call pager: (S31) 64S-2177. 


Signature of Researcher 

Date 



Signature of Voluntary Participant 

Date 
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5. 


MAPS TO NPS 



* Enter the 1St and Sloat 

* Follow perimeter road: Morse Rd. turns into 
Lake Del Monte, turns into Cunningham Rd. 

* Turn Right on Bouldry Road and look for 
parking. Parking is very difficult on NPS - 
allow 15 min. just to find a space! ^ 

* Lab is in Glasgow Hall rm 221, ph 656-3134. 

* Capt Takagi cell phone 236-6306. 

l200m I K ^ 
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6 . 


PARTICIPANT ID FORM AND ID CARDS 


FOR OFFICIAL USE ONLY 

This document contains personally identifying research information and 
may only be viewed by authorized personnel associated with this research. 


Participant ID # 


Date 



Please keep your participant ID # with you at all times until the conclusion 
of the study. 

The information below will be used only if there is a need to contact you 
after the conclusion of this study. Special care will be taken to preserve 
anonymity during all data collection and analysis. Any comments or opinions you 
express during this study will not be attributed to you in any manner. Written 
documentation will be identified by your Participant ID number only, and faces 
and voices will be obscured on video recordings when possible. No information 
which could identify you as a participant will be made accessible to your military 
unit, your supervisors, or to the public without your consent. 


Last name 


First name, middle 
initial 


Rank 


Branch of service 


Unit or 
organization 
POC information 

please give a mailing 
address, and phone number that 
we could use to contact you in 
the future, if necessary 


emaii address 



FOR OFFICIAL USE ONLY 

This document contains personaliy identifying research information and 
may only be viewed by authorized personnel associated with this research. 
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Research Participant 


o 

Naval Postgraduate School 

Applied Warfighter Ergonomics 

Glasgow Hall room 221 
(831) 656-3134 

O 

AWE 

Applied Weiiighler Ergonomics 

Participant ID# 

AWE 

Applied Warfighter Ergonomics 


Research Participant 


o 

Naval Postgraduate School 

Applied Warfighter Ergonomics 

Glasgow Hall room 221 
(831) 656-3134 

O 

AWE 

Applied Warfighter Crgooomics 

Participant ID# 

AWE 

Applied Warfighter Ergonomics 


Research Participant 


o 

Naval Postgraduate School 

Applied Warfighter Ergonomics 

Glasgow Hall room 221 
(831) 656-3134 

O 

AWE 

Applied Warfighter Ergonomics 

Participant ID# 

AWE 

Applied Warfighter Ergonomics 


Research Participant 


O 

Naval Postgraduate School 

Applied Warfighter Ergonomics 

Glasgow Hall room 221 
(831) 656-3134 

O 

AWE 

Applied Warfighter Ergonomics 

Participant ID# 

1 - 1 

AWE 

Applied Warfighter Ergonomics 


Research Participant 


O 

Naval Postgraduate School 

Applied Warfighter Ergonomics 

Glasgow Hall room 221 
(831) 656-3134 

o 

AWE 

Applied Warfighter Ergonomics 

Participant ID# 

1-1 

AWE 

Applied Warfighter Ergonomics 


Research Participant 

Naval Postgraduate School 
Applied Warfighter Ergonomics 
Giasgow Mali room 221 
(831) 656-3134 

Participant iD# 


Research Participant 

Naval Postgraduate School 
Applied Warfighter Ergonomics 
Giasgow Hall room 221 
(831) 656-3134 

Participant iD# 


Research Participant 

Naval Postgraduate School 
Applied Warfighter Ergonomics 
Glasgow Hall room 221 
(831) 656-3134 

Participant iD# 


Research Participant 

Naval Postgraduate School 
Applied Warfighter Ergonomics 
Glasgow Hall room 221 
(831) 656-3134 

Participant iD# 


Research Participant 

Naval Postgraduate School 
Applied Warfighter Ergonomics 
Glasgow Hall room 221 
(831) 656-3134 

Participant iD# 






















APPENDIX I. INSTITUTIONAL REVIEW BOARD FORMS 



Nita Lewis Miller, Ph.D* 
Operations Research DepartmeDt 
Glasgow Hall 

Naval Postgraduate School 
Monterevj California 93943 


331-656-2281 
DSN: 756-2281 
Fax: 831-656-2595 
Q Imi 11 e r@n p s. n mi I 


To: Protection of Human Subjects Committee 

Subject: Application for Human Subjects Review for Evaluating the Human Factors of Military 

Handheld Computing Devices 

1. Attached is a set of documents outlining surveys, direct observation and recording, 
interviews, after-action reports, and focus group studies to be conducted over the next year. 
Some studies will be conducted in conjunction with the Tactical Network Topology (TNT) 
project at NPS, Inter-4 Corporation of San Francisco, California, Exponent Corporation of 
Menlo Park, California, the United States Army Special Operations Command (USASOC), 
and the United States Marine Corps, 

2. We are requesting approval of the described experimental protocol. An experimental outline 
is included for your reference that describes the methods and measures we plan to use. 

3. We include the consent forms, privacy act statements, all materials and forms that a subject 
will read or fill-out, and the debriefing forms (if applicable) we will be using in the 
experiment. 

4. We understand that any modifications to the protocol or instruments/measures will require 
submission of updated IRE paperwork and possible re-review. Similarly, we understand that 
any untoward event or injury that involves a research participant will be reported 
immediately to the IRB Chair and NPS Dean of Research. 



Nita Lewis Miller, PhD 


Koichi Takagi, Capt USMC 
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APPLICATION FOR 
HLIVIAN SUBJECTS REVIEW (HSR) 


HSR NUMBER 


(to be assigned) 


PRENJCIPAL INVESTIGATOR(S) (Full Name, Code. Telephone) 

Nita Lewis Miller, Ph.D., CODE 61 83I-656-228I DSN: 756-2281 Fax: 831-656-2595 
Koichi Takagi, Capt, USMC, CODE 61 831-656-2281 DSN: 756-2281 Fax: 831-656-2595 


APPROVAL REQUESTED [ X ] New [ ] Renewal 


LEVEL OF RISK ( ] Exempt ( X ] Minimal [ ] More than Minimal 
Justification: 


WORK WILL BE DONE IN (Site/Bldg/Rm) 

Glasgow' 221, Camp Roberts, California, Fort 

Ord, California, and other field locations 

ESTIMATED NUMBER OF DAYS TO COMPLETE 

180 days 

MAXIMLT^ NUMBER OF SUBJECTS 

100 

ESTIMATED LENGTH OF EACH SUBJECT’S 
PARTICIPATION 

cst. total 6 hrs, 2-4 hrs per day, in a 2 week period 


SPECIAL POPULATIONS THAT WILL BE USED AS SUBJECTS 

( ] Subordinates [ ] Minors [ X ] NPS Students [ ] Special Needs (e.g. Pregnant women) 

Specify safeguards to avoid undue influence and protect subject’s rights: 

Special care will be taken to preserve anonymity in all data collection. Written documentation will be 
identified by number only, and faces and voices may be obscured on video recordings. _ 

OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 

Tactical Network Topology (TNT) project at NPS, Inter-4 Corporation of San Francisco, California, 
Exponent Corporation of Menlo Park, California, the United States Army Special Operations 
Command (USASOC), and the United States Marine Corps. _ 

TITLE OF EXPERIMENT AND DESCRIPTION OF RESEARCH (attach additional sheet if needed). 
Army Rangers, Special Forces team members, and other DoD enlisted members and officers, will be 
field testing new' handheld computers (including Intcr-4's Tacticomps, NavBoards and Exponent ARC 
PDAs) and the Tactical Netw'ork Topology (TNT) system. Other handheld systems may be tested in 
similar environments, as available. A short pre-exercise will be given to assess each participant's 
familiarity and comfort in using current technology. During- and post-exercise surveys, interview's, and 
focus groups will assess the performance and usability of the equipment. Participants' interactions w'ith 
the equipment may be recorded (audio and video) during field exercises and in laboratory settings, and 
detailed records (behavioral coding) made of their acti\ilies and reactions. The usability of military 
handheld electronic devices, and the factors influencing the acceptance and adoption of those devices, 

w'ill be analyzed as a product of this research. _ 

I have read and understand NPS Notice on the Protection of Human Subjects. If there are any changes 
in any of the above information or any changes to the attached Protocol, Consent Form, or Debriefing 
Statement, I wilTs^end the e^erimenumtil I obtain new Committee approval. 

SIGNATURE _ DATE 

SIGNATURE // _ DATE 
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APPENDIX J. SOFTWARE AND HARDWARE CONFIGURATION 


1. NOLDUS "THE OBSERVER" SOFTWARE 

Observation files were named using this convention: 

R_ 

The first letter after "R" is the device tested, either A-D for the RPDAs, or Z 
for the conventional equipment. 

The next four digits are the participant's ID, in the range 0500-0599 for this 

study. 

The last two character's are the coder's initials, either "KT" for Koichi 
Takagi or "KB" for Katie Buettner. 
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X-KEYS MACRO KEYBOARD 


In addition to the key assignments shown below, the jog dial (inner 
spinning wheel) was set to FFWD (jog right) and REW (jog left). 



X-keys Macro Maker - Takagi RPDA study conf.xk4 Editor, Green layer 


Key Type Comment 

Key Type Comment 

Key Type Comment 

0 Macro edit obs. 

30 Macro OFF 2c plot rte 

60 Macro 

1 Macro « REW 

31 Macro tactical 

61 Macro -4x 

2 Macro FFWD » 

32 Macro admin 

62 Macro -2x 

3 Empty 

33 Macro comment 

63 Macro -lx 

4 Macro PLAY > 

34 Empty 

64 Macro -1/2x 

5 Macro PAUSE 

35 Empty 

65 Macro -1/5x 

6 Macro STOP 

36 Macro ON 2a plot rte 

66 Macro -1/25x 

7 Macro Quick Rev 

37 Macro OFF 2a plot rte 

67 Empty 

8 Empty 

38 Macro ON 3c read grid 

68 Macro 1/25x 

9 Empty 

39 Macro OFF 3c read grid 

69 Macro 1/5 

10 Empty 

40 Macro ON 4c brg rng 

70 Macro 1/2 

11 Empty 

41 Macro OFF 4c brg rng 

71 Macro lx 

12 Empty 

42 Macro ON 5c send msg 

72 Macro 2x 

13 Empty 

43 Macro OFF 5c send msg 

73 Macro 4x 

14 Empty 

44 Macro ON 6c rev msg 

74 Macro FFWD 

15 Empty 

45 Macro OFF 6c rev msg 

75 Macro Play 

16 Empty 

46 Macro ON 7c read back 

76 Macro 

17 Empty 

47 Macro OFF 7c read back 

77 Macro 

18 Empty 

48 Macro ON 3a read grid 

78 Macro 

19 Empty 

49 Macro OFF 3a read grid 

79 Macro 

20 Macro ON 1c comm chk 

50 Macro ON 4a brg rng 

80 Macro 

21 Macro OFF 1c comm chk 

51 Macro OFF 4a brg rng 

82 Empty 

22 Macro slip 

52 Macro ON 5a send msg 

83 Macro Still 

23 Macro mistake 

53 Macro OFF 5a send msg 

84 Macro 

24 Macro device error 

54 Macro ON 6a rev msg 

85 Macro 

25 Macro interface error 

55 Macro OFF 6a rev msg 

86 Macro 

26 Macro equipment handling error 

56 Macro ON 7a read back 

87 Macro 

27 Macro ON la comm chk 

57 Macro OFF 7a read back 

88 Macro 

28 Macro OFF la comm chk 

58 Macro FFWD 

89 Macro Play 

29 Macro ON 2c plot rte 

59 Macro RWD 
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APPENDIX K. DATA 


1. FOCUS GROUP NOTES 

RPDA Focus Group 

Ad-hoc, non-attributional military RPDA Focus Group conducted 1600 - 1730 on 
22 Feb 2005 at Camp Roberts, CA. Facilitator: Captain Koichi Takagi, USMC, 
student researcher at the Naval Postgraduate School 


Participants: 

Three senior Special Forces staff NCOs participating in a military PDA evaluation 
exercise. 

One representative from USASOC program office (but not the Mobile Access 
Infrastructure program office responsible for military PDAs). 

One employee of an RPDA development company. 


Purpose: to determine possible requirements and capabilities for handheld 
military computing devices (military PDAs). A “brainstorming” environment during 
the focus group was sought. Ideas were encouraged regardless of current 
feasibility or real applicability. The following is a list of topics discussed, including: 
possible operational requirements for military PDAs, general and specific 
functionality that military PDAs might have, possible uses for and tactical 
employment of military PDAs, and random comments. Some participants had not 
used military or civilian PDAs to any great extent, some were unfamiliar with the 
military tasks discussed. Non-attribution was emphasized, and it was understood 
that the discussion would not generate obligations by the contractor or the 
government. 
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Four critical requirements defined as broad functionality required of a military 
PDA: 

Text messaging, Voice over IP (VOIP) capability, Moving video and still imagery 
dissemination, and GPS - mapping - situational awareness (SA) - “Blue Force 
Tracker (BFT)” capability. 

Text messaging 

ease of input (keyboard, on-screen keyboard, stylus, or other) 

speed and accuracy of input are crucial - must test error rate and speed 

input must be reliable under a wide range of operating environments (bumpy 
vehicle, day, night, one handed, etc.) 

voice recognition? Again, must be reliable under combat conditions - injury, 
illness, extreme background noise 

cut & paste between messages sent and/or received 

word wrap so that whole message is always visible, rather than a scrolling single 
line 

ability to change text size for different conditions (bumpy driving, day, night, etc.) 

lexicon of standard military terms and abbreviations 

auto-correct and spell check based on above lexicon 

auto-complete - this might be bad if it made dangerous errors 

recent word history 

cues for message received - audio, visual, vibrate? Be able to turn off alerts 
message sent confirmation (is network up? Did I do it right?) 
delivery confirmation (did they get it?) 
keep chat open but minimized 

advantage - text chat is 1000 times less bandwidth than voice. 

Things we would or could do with text messaging: 

• Lists of info to save for later - convoy routes, admin details, fragOs, op 
order 

• Distribution groups for an entire unit 

• Add annotation to pictures, describe details 

• Flag messages as important for later review / action 
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Retrieve past messages, edit and send on to others 




How would we do this without a text-messaging military PDA 
Other data means - SMEARS? Mini laptop/palmtop VDC card ViaSat 


Voice over IP (VOIP) 

Important: quality of transmission, reliability, clarity (comparable to current 
systems) 

Full duplex? Don’t see need 
Record/playback capability 

Probably won’t replace any current voice radio systems (MBTRs, SINCGARS, 
etc.) for a long time, so would be an additional piece of gear to carry just for voice 
comm. Because of this, is VOIP really a pressing need? We will still use many 
other ways to talk. 

Used for: 

• Emergency / backup long-range comm, reach-back to higher command 

• Flexibility, alternate means 

• Collaboration while viewing images (Groove) 

• Mobile whiteboard - annotate images, drawings, while talking about it 
Video and Still imagery dissemination 

Requirements: minimum 5 fps, minimize jitter and data loss 

Network needs to be able to provide a minimum of 3 simultaneous video streams 
(video sources) 

Quality is affected by sensor capabilities (color, contrast, zoom) 

Current streaming video is “quarter VGA” (QVGA) = 320x240 pixels (similar to 
VHS tape or broadcast TV) 

Would like a local buffer / save capability beyond just streaming so we can 
playback, rewind, etc. 

Search & Rescue “panic button” to show unit what’s happening if I go down. 

File type compatibility has not been an issue 

Different client users involved: dedicated intel analysts (formally school-trained or 
not), and untrained regular operators. These different users have different 
expectations and capabilities when given the same image - should test 
separately. 
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We can measure things like frame rate, number of pixels, network jitter, etc., but 
the final measure of success is subjective - can the user correctly interpret the 
video presented? 


Mapping, GPS, situational awareness. Blue Force Tracker (BFT) 

System should replace BFT, SA tools, MTS 
Should completely replace PLGRS 

BFT / SA must have restricted lists for SOF teams - not everyone is allowed to 
see where they are 

Mapping - how does interpreting an on-screen, 2inch by 3inch map compare 
mentally to reading a paper map? 

Paper provides far higher resolution (2400dpi+ compared to 72dpi on screen), 
larger area, quicker and more intuitive manipulation, easier stowage. 

Computer capability - have a whole map library in your pocket, from 1:250,000 
down to room-by-room blueprints. 

Should provide all standard map overlay information like routes, NAIs, RFAs, etc. 
Filters for just the units and info I want to know. 

Secure comm, critical 

Need to have some positive indicator that the units and overlay information 
shown is reliable or not, is old or new, and how precise it is. 

Time of last update shows as a pop-up? 

Icons change color as they “age”? 

Should map also show when you are off the network, like C2PC 

If there is nothing shown on my map, am I missing info, off the net, or is the area 
truly clear? 

Indications of what we know, what we don’t know, etc. 

Ultimate measure of success: user’s ability to understand the info presented and 
translate it into ground truth. 

Everyone looks at a map and a situation differently. Must be able to customize 
view for each individual operator. 

Common iconology 

Classic icons used on paper don’t always translate well to lo-res screen. 
Example - checkpoint symbol is usually very hard to see on screen because it’s 
so small. Unit symbols are hard to interpret well. Maybe it’s time for new icons for 
digital maps. 
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2. SURVEY DATA: INDIVIDUAL COMMENTS 
RPDA A - Comments from Handheld Device Evaluations 

6. Satisfaction With Text Messages Sent 

(501) 4- easy to hit wrong key while typing 

(502) 4- keypad letters small, but usable 

(506) 4- voice recognition or a small keypad would make entries easier than with 
the stylus; transmission was easy to effect 

8. Satisfaction With Text Messages Received 

(502) 5- would prefer notification when message received 
(506) 5- easy to read 

17. Satisfaction With Mapping Functions 

(501) 3- finding labeled places was easy, but finding other locations was hard 

(502) 4- easy to view self and others; some symbols hard to interpret (road, 
airport) 

(506) 2- small screen made it difficult to orient self quickly 

18. Satisfaction With SA Toois 

No comments 

20. Satisfaction With Stylus 

(501) 1- hard to use the stylus to scroll around map 

(505) 5- touch keypad small and possible to make mistakes, but overall good 
feature 

(506) 4- stylus ok, but would prefer keypad or number pad like on a cell phone 

26. Ability to Use Personal Weapon 

(501) 1- do not see how you could operate Ml6 and RPDA 

(502) 1- only able to hold weapon with one hand 

(505) 2- hard to hold RPDA and maintain good grip on weapon 

(506) 2- cumbersome; RPDA's wires got tangled with weapon sling 

28. Overall Satisfaction With Reiiabiiity 

(501) 2- reliable for sending messages, but some issues with stylus navigation of 
the map 

(502) 5- no malfunctions; confident in RPDA's calculations 
(506) 5- no errors or malfunctions 

30. Overall Satisfaction With System 

(501) 2- stylus tricky to use in map mode and RPDA gear bulky and harder to 
move with 
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(502) 5- helpful in eliminating human error in calculating brg/rng; text messaging 

quieter than radio for covert situations 

(506) 4- effective communication, but hard to use quickly 

31. Impression of Ease of Learning and Use 

(502) 5- extremely easy 
(506) 5- easy to operate 

32. Free Comments 

(501) separate backpack and wires make RPDA A less desirable 

(502) helpful if equipment manufactured to hold device, so when not in use 
hands could be free for weapon; clearer resolution on some map features; 
notification of received message 

(505) no big glitches; makes navigation/communication tasks a little easier 

(506) more effective if attached to forearm or hand to allow simultaneous weapon 
handling 


RPDA B - Comments from Handheld Device Evaluations 

6. Satisfaction With Text Messages Sent 

(501) 4- easy to hit wrong key, wasting time 

(502) 1- hard to hit correct letters because of irresponsiveness of stylus 
(505) 4- had to push down hard with stylus, which led to double typing a letter 

8. Satisfaction With Text Messages Received 

No comments 

17. Satisfaction With Mapping Functions 

(501) 2- hard to read names on map 

(502) 3- hard to pull up coordinates because of stylus 

18. Satisfaction With SA Toois 

No comments 

20. Satisfaction With Styius 

(501) 4- stylus was fairly capable 

(502) 1- horrible responsiveness, cumbersome on every level, especially when 
using keypad 

(505) 4- had to push hard 

26. Ability to Use Personal Weapon 

(501) 1- could not aim or shoot weapon while operating RPDA, in terms of 

mechanisms and concentration 

(505) 3- hard to operate simultaneously 
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28. Overall Satisfaction With Reiiabiiity 

(501) 4- easy to hit wrong key while typing 

(502) 1- stylus 

30. Overall Satisfaction With System 

(501) 4- liked size; liked receive message function and reading old messages, 
both sent and received 

(502) 1- stylus 

31. Impression of Ease of Learning and Use 

(502) 5- easy to learn 

32. Free Comments 

(501) hard to type and concentrate on event in the scenario; system better for 
reconnaissance or sniper mission to quietly give reports/receive orders w/o radio 
noise; in high speed situations, a radio is preferable; finding grid coordinates, 
distance, direction, of location in reference to oneself works well, but it would be 
better if one could also find distance, direction between two points independently 
of one's own location 

(502) too bulky and stylus too difficult to use 

(505) only flaw was stylus—pushing down harder made it difficult 

RPDA D - Comments from Handheld Device Evaluations 

6. Satisfaction With Text Messages Sent 

(528) 5- used chat slang for brevity 

8. Satisfaction With Text Messages Received 

No comments 

17. Satisfaction With Mapping Functions 

No comments 

18. Satisfaction With SA Toois 

(528) 4- would like to see current (?) data displayed—panic button for key data 

20. Satisfaction With Stylus 

(528) 4- alternate means if lost? damage to screen potential 

26. Ability to Use Personal Weapon 

(515) 2- requires weapon to be slung to operate 

(527) N/A- M9 or M4 more suited for use with this device 

(528) none- difficult w/o a drop capability 
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28. Overall Satisfaction With Reiiabiiity 

(527) 4- comms piece came loose at first (system crashed) 

30. Overall Satisfaction With System 

(528) 4- software should be upgraded 

31. Impression of Ease of Learning and Use 

(528) 5- would use in combat with suggestions made 

32. Free Comments 

No comments 
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SURVEY DATA: INDIVIDUAL NUMERICAL RESPONSES 


:S1 
TES2 
TES3 
TES4 
TES5 
TES6 
TES7 
TES8 
TES9 
TES10 
TES11 
TES12 
TES13 
TES14 
TES15 
TES16 
TES17 
TES18 
TES19 
TES201 
TES202 
TES203 
TES20 4 
TES205 


TES21 

TES22 

TES23 

TES24 

TES25 

TES26 

TES27 

TES28 

TES29 

TES30 

TES31 

TES32 

TES38 

TES34 

TES35 

TES36 

TES37 

TES38 

TES39 

TES40 

TES41 

TES42 

TES43 

TES44 

TES45a 

TES45b 

TES45C 

TES45d 

TES45e 

TES45f 

TES45g 

TES45h 



RPDA-D 





tisted in order tested 

-> 


Participant Number 

0513 

0514 

0515 

0528 

0527 

have cell phone (yes/no) 

1 

1 

0 

1 

0 

talk on phone (hours) 

9 

5 

0 

5 

0 

use text messaging (hours) 

2 

2 

0 

1 

0 

customize cell phone (hours) 

1 

1 

0 

0.2 

0 

play games or cell phone (hours) 

1 

2 

0 

0 

0 

phone numbers stored in cdl phone 

40 

75 

0 

100 

0 

PDA use 

2 

2 

2 

2 

1 

PDA stylus use 

3 

3 

3 

3 

1 

videogame pay (hours) 

7 

7 

2 

0 

0 

oriire video games (yes/no) 

□ 

1 

1 

0 

0 

home computer video games (yes/no) 

1 

1 

1 

□ 

0 

video game system video games (yes/no) 

1 

1 

1 

0 

0 

list favorite games 

* 

A 

* 


A 

has wo(k corrputerforown use (yes/no) 

0 

0 

0 

1 

1 

has shared work conripuler (yes/no) 

1 

1 

0 

0 

0 

almosi never use computer at work [yes/no) 

□ 

0 

1 

□ 

0 

use computer in living quarters (yes/ro) 

1 

1 

1 

1 

1 

use pifclic computers (yes/no) 

1 

0 

1 

0 

0 

use severaD programs at same lime [yes/no) 

1 

1 

1 

1 

1 

computer probem: a. slop 

5 

5 

2 

4 

5 

computer probem: b read manual 

3 

3 

5 

5 

4 

computer probem: c read help 

2 

2 

3 

3 

2 

computer probem: d ask for help 

4 

4 

4 

2 

3 

computer probem: e keep trying 

1 

1 

1 

1 

1 


RPDA-D 





itsted in order tested 

-> 


Participant Number 

0513 

0514 

0515 

052S 

0527 

use a computer 
*use email (hrs per week) 

5 

200 

20 

200 

100 

'use email attachments (hrs per week) 

1 

100 

2 

50 

50 

'use online chat cr IM (hrs per week) 

1 

1000 

0 

0 

0 

'use internet to find information (hrs per week) 

1 

50 

15 

50 

20 

'shop online (hrs per MONTH) 

0 

5 

1 

2 

0 

'use orpine banking [hrs per MONTH) 

1 

20 

0 

2 

3 

trust cornputers (yes/no) 

1 

1 

0 

1 

1 

expenence with radio 

2 

1 

1 

4 

4 

expenence with GPS 

1 

1 

1 

5 

4 

expenence with topographical map 

3 

1 

2 

5 

4 

experience with military compass 

2 

4 

2 

5 

4 

experience with FBCB2 or BFT 

0 

0 

0 

4 

0 

experience with Tacticomp 

0 

0 

0 

0 

0 

experience with CDA or other RPDA 

□ 

0 

0 

5 

0 

age 

19 

1S 

19 

31 

27 

gender 

M 

M 

M 

M 

M 

pay grade 

E2 

E2 

E3 

03 

E6 

branch of service 

USMC USMC USMC 

USA 

USA 

years enlisted service 

1 

1 

1 

0 

9 

years commissioned service 

0 

0 

0 

9 

0 

designator. MOS, cr rating 

2600 

2600 

2670 

35D 

98G 

level of education (1-5) 

2 

2 

3 

4 

3 

current assignment 

* 

* 

* 

* 

* 

special sc hods: Ranger 

□ 

0 

0 

□ 

0 

special sc hods: Pathfinder 

□ 

0 

0 

□ 

0 

special sc hods: EIB 

0 

0 

0 

0 

0 

special sc hods: Air Assad t 

0 

0 

0 

0 

1 

special sc hods: NOO Course 

0 

0 

0 

0 

1 

special schods: SNCO Course 

□ 

0 

0 

□ 

0 

special schods: Computer Training 

□ 

0 

0 

□ 

0 

special schods: HALO 

□ 

0 

0 

□ 

0 
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Takagi Thesis Raw Data 


Participant Number 

RPDA-D 

listed in order tested 
0513 0514 0515 

0528 

□527 


Device used this trial 

4 

4 

4 

4 

4 


Order 

1 

1 

1 

1 

1 

HDE1 

location 






HDE2 

dale 






HDE3 

description of scenario 






HDE4 

description offeahjres 






HDE5a 

#toxt messages sent (altemplecq 

3 

3 

4 

5 

5 

HDE5b 

#t6xt messages sent (successfiJ} 

3 

3 

3 

5 

5 

HDE6 

satisfaction iftith text messages sent 

A 

5 

4 

5 

5 

HDE/a 

#text messages received (attempted) 

3 

3 

3 

3 

3 

HDE/b 

#text messages received (successfiJ} 

3 

3 

3 

3 

3 

HDES 

satisfaction iMth to^it rec^ved 

5 

5 

5 

5 

5 

HDE9 

satisfaction vMh comm and network 

4 

5 

4 

5 

4 

HDE10 

satisfaction with reliability of comm and network 

5 

5 

4 

5 

5 

HDE11 

satisfaction with still images 

3 

3 

4 

A 

5 

H DE12a # riKwing videos viewed (atte mpled) 

0 

6 

0 

0 

0 

HDE12b # mwing videos viewed (successful) 

6 

6 

0 

0 

0 

HDE13 

satisfaction with quality ct moving video 

5 

5 

0 

0 

0 

H DEI 4a 

# audio i os listened to (attempted] 

6 

6 

0 

0 

0 

HDEI 4b 

# a udio i os 1 iste ned to (successfil) 

6 

6 

0 

0 

0 

HDE15 

satisfaction with quality of audo files 

5 

5 

0 

0 

0 

HDE16a 

#of uses of moping functions (attempted] 

5 

5 

3 

3 

3 

H DE16b # Of uses of mappi ng fuTKIons (successfU ) 

5 

5 

3 

3 

3 

HDE17 

satisfaction with mapping functions 

4 

4 

4 

4 

5 

HDE18 

satisfaction with SA fools 

3 

3 

0 

4 

0 

HDE19a 

#of uses of stylus (attenriptec^ 

10 

10 

7 

10 

20 

H DE19b # Of uses of slyl us (successf J ] 

10 

10 

7 

10 

20 

HDE20 

satisfaction with stylus 

3 

4 

4 

4 

5 

HDE21a 

# of uses of buttons (attempted) 

0 

0 

0 

0 

0 

HDE21b 

# of uses of buttons (successfU) 

□ 

0 

0 

0 

0 

HDE22 

satisfaction with buttons 

□ 

0 

0 

0 

0 

HDE23 

satisfaction with one harxled operation 

□ 

0 

0 

2 

0 

HDE24 

satisfaction with removing device to use it 

0 

0 

0 

0 

0 

HDE25 

satisfaction with stowing device 

0 

0 

0 

0 

0 

HDE26 

ability to use personal weapon 

2 

0 

2 

0 

0 

HDE27 

ability to read the screen 

□ 

0 

0 

0 

0 

HDE2S 

overall satis^ction with reliebiity 

4 

4 

5 

0 

4 

HDE29 

overall satisfaction with ruggedness 

3 

3 

0 

4 

0 

HDE30 

overall satisfaction with system 

A 

4 

4 

4 

5 

HDE31 

impressEon of eased learning and use 

3 

4 

4 

5 

5 

HDE32 

free comments 



230 





RPOA-A 


Participant Number 

0502 

0506 

0501 

0503 

0505 

0504 

TES1 

have cell phone (yes/no) 

0 

1 

0 

1 

1 

1 

TES2 

talk on phone (hours) 

0 

7 

0 

14 

5 

5 

TES3 

use text messaging (hours) 

□ 

0 

0 

2 

2 

0 

TES4 

cnstomizecQll phone (hours) 

□ 

0 

0 

1 

1 

1 

TES5 

play games or cell phone (hours) 

□ 

1 

0 

0 

1 

0 

TES6 

phone numbers stored in cdl phone 

0 

40 

0 

70 

30 

20 

TES7 

PDA use 

1 

2 

1 

2 

1 

1 

TES8 

PDA stylus Lise 

1 

3 

1 

3 

1 

1 

TES9 

video game play (hours] 

10 

0 

6 

50 

2 

3 

TES10 

oriire video games [yes/no) 

0 

0 

G 

1 

1 

1 

TES11 

home co^Tf^u[e^ video games [yes/no) 

0 

0 

1 

1 

1 

1 

TES12 

video game system video games (yes/no) 

1 

1 

1 

1 

1 

0 

TES13 

list favorite games 

* 

* 

* 

* 

« 

* 

TES14 

has work computer for own use (yes/no) 

□ 

1 

0 

1 

1 

0 

TES15 

has shared work conrpjler (yes/no) 

0 

0 

1 

1 

1 

0 

TES16 

almosi never use computer at work [yes/no) 

□ 

0 


0 

0 

1 

TES17 

use computer in living quarters (yes/no) 

1 

0 


□ 

0 

1 

TES18 

use public computers (yes/no) 

1 

1 


1 

1 

1 

TES19 

use severaJ programs at same lime (yes/no] 

0 

1 


1 

1 

1 

TES20 1 computer proberr: a stop 

4 

4 

5 


5 

4 

TES20.2 computer probem: b. Fead manual 

5 

5 

4 


3 

5 

TES20 3 computer probem: c. read hdp 

3 

2 

1 


2 

3 

TES20 4 computer probem: d ask for hdp 

2 

3 

3 

2 

4 

1 

TES20 5 computer probem: e keep trying 

1 

1 

2 

1 

1 

2 



RPDA-A 






Participant Number 

0502 

0506 

0501 

0503 

0505 

0504 

TES21 

use a computer 



4 




TES22 

*use email (hrs per week) 

1 

10 

2 

140 

20 

20 

TES23 

'use email attachments (hrs per week) 

□ 

1 

0 

50 

10 

5 

TES24 

'use online chait cr IM (hrs per week) 

□ 

0 

0 

7 

15 

5 

TES25 

*useinlerr>ettofind informalion [hrs per week) 

1 

70 

1 

7 

20 

20^ 

TES26 

'shop online (hrs per MONTH) 

0 

4 

3 

4 

3 

0 

TES27 

'use online banking [hrs per MONTH) 

0 

2 

0 

2 

5 

10 

TES28 

trust corriputers (yes/no) 

1 

1 

1 

1 

1 

1 

TES29 

experience with radio 

1 

1 

1 

0 

1 

0 

TES30 

experience with GPS 

□ 

0 

1 

□ 

1 

0 

TES31 

experience with topographical map 

3 

1 

2 

3 

2 

1 

TES32 

experience with military compass 

3 

1 

2 

3 

2 

1 

TES33 

experience with FBCB2 or BFT 

0 

0 

1 

0 

G 

G 

TES34 

experience with Tacticomp 

0 

1 

1 

0 

0 

0 

TES35 

experience with CDA or other RPDA 

□ 

0 

1 

□ 

0 

0 

TES36 

age 

24 

36 

20 

18 

19 

21 

TES37 

gender 

M 

M 

M 

M 

M 

F 

TES38 

paiyyade 

E4 

E4 

E3 

E2 

E2 

E3 

TES39 

branch of service 

USA 

USA 

USA 

USA 

USA 

USA 

TES40 

years enlisted service 

1 

1 

1 

1 

1 

1 

TES41 

years commissioned service 

0 

0 

0 

0 

0 

0 

TES42 

designator, MOS, cr rating 

93X 

97E 

98 

98G 

98X 

98G 

TES43 

level, of education (1-5) 

4 

4 

3 

2 

3 

3 

TES44 

current assignment 

* 



* 

A 


TES45a 

special schcds: Ranger 

0 

0 

0 

0 

0 

0 

TES45b 

special sc hods: Pathfinder 

0 

0 

0 

0 

0 

0 

TES45C 

special schods: EIB 

□ 

0 

0 

□ 

0 

0 

TES45d 

special schods: Air Assault 

0 

0 

0 

0 

0 

0 

TES45e 

special schcds: NOO Course 

0 

0 

0 

0 

0 

0 

TES45f 

special schods: SNOO Course 

□ 

0 

0 

□ 

0 

0 

TES45g 

special schods: Corrputer Training 

□ 

0 

0 

□ 

0 

0 

TES45h 

special schods: HALO 

□ 

0 

0 

□ 

0 

0 
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Takagi Thesis Raw Data 


Partici|iant Number 

RPDA-A 

0502 0506 

□501 

0503 

□505 

□504 


Device used this trial 

1 

1 

1 

1 

1 

1 


Order 

1 

1 

2 

2 

1 

2 

HDE1 

location 







HDE2 

dale 







HDE3 

description of scenario 







HDE4 

description offeahjres 







HDE5a 

#toxt messages sent (altemplecq 

5 

2 

5 


4 


HDE5b 

#t6xt messages sent (successfiJ} 

5 

2 

5 


4 


HDE6 

satisfaction iftith text messages sent 

4 

4 

4 


4 


HDE/a 

#text messages received (attempted) 

5 

3 

5 


4 


HDE/b 

#text messages received (successfiJ} 

5 

3 

5 


4 


HDES 

satisfaction iMth te^it mess^es rec^ved 

5 

5 

4 


5 


HDE9 

satisfaction vMh comm and network 

4 

4 

4 


4 


HDE10 

satisfaction with reliabiliEy of comm and network 

5 

2 

4 


4 


HDE11 

satisfaction with still images 

4 

4 

1 


3 


H DE12a # riKwing videos viewed (atte mpled) 

1 

6 

0 


6 


HDE12b # rnwing videos viewed (suocessflil) 

1 

6 

0 


6 


HDE13 

satisfaction with quality of moving video 

5 

5 

0 


5 


H DEI 4a 

# audio ies listened to (attenpted] 

1 

7 

0 


6 


HDEI 4b 

# audio ies listened to (successfti) 

1 

7 

0 


6 


HDE15 

satisfaction with quality of audo files 

5 

5 

0 


5 


HDE16a 

#of uses of moping functions (attempted] 

3 

4 

3 


5 


H DE16b # Of uses of mappi ng fuTKIons (successlU ) 

3 

4 

3 


5 


HDE17 

satisfaction with mapping functions 

4 

2 

3 


4 


HDE18 

satisfaction with SA lools 

□ 

3 

4 


5 


HDE19a 

#of uses of stylus (attenriptec^ 

75 

7' 

8 


10 


H DE19b # Of uses of slyl us (successf J ] 

75 

7 

8 


10 


HDE20 

satisfaction with stylus 

5 

4 

1 


5 


HDE21a 

# of uses of buttons (attempted) 

0 

0 

0 


1 


HDE21b 

# of uses of buttons (succksM) 

□ 

0 

0 


1 


HDE22 

satisfaction with buttons 

□ 

0 

0 


3 


HDE23 

satisfaction with one harxled operation 

2 

3 

1 


5 


HDE24 

satisfaction with removing device to use it 

0 

2 

0 


3 


HDE25 

satisfaction with stowing device 

0 

0 

3 


3 


HDE26 

abilily to use peisdid msipm 

1 

2 

1 


2 


HDE27 

ability to read the screen 

5 

0 

0 


4 


HDE2S 

overall satis^ction with holjiiblity 

5 

5 

2 


5 


HDE29 

overall satisfaction with ruggedness 

5 

0 

0 


5 


HDE30 

overall satisfaction with system 

5 

4 

2 


5 


HDE31 

impression of ease of learning and use 

5 

5 

5 


5 


HDE32 

free comments 

* 


* 
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Takagi Thesis Raw Data 


Partici|>ant Number 

RPDA-B 

0502 0506 

□501 

0503 

□505 

□504 


Device used this triaf 

2 2 

2 

2 

2 

2 


Order 

2 2 

1 

1 

2 

1 

HDE1 

location 






HDE2 

dale 






HDE3 

description of scenario 






HDE4 

description offeahjres 






HDE5a 

#toxt messages sent (altemplecq 

1 

6 


1 


HDE5b 

#t6xt messages sent (successfiJ} 

1 

6 


1 


HDE6 

satisfaction iftith text messages sent 

1 

4 


4 


HDE/a 

#text messages received (attempted) 

1 

6 


1 


HDE/b 

#text messages received (successfiJ} 

1 

6 


1 


HDES 

satisfaction iMth te^tt mess^es rec^ved 

5 

4 


5 


HDE9 

satisfaction vMh comm and network 

2 

4 


4 


HDE10 

satisfaction with reliability of comm and network 

5 

5 


5 


HDE11 

satisfaction with still images 

3 

2 


4 


HDE12a 

# riKwing videos viewed (attempted] 

0 

0 


1 


HDE12b 

# rawing videos viewed (suooessftd) 

0 

0 


1 


HDE13 

satisfaction with quality of moving video 

□ 

0 


4 


H DEI 4a 

# audio ies listened to (attempted] 

□ 

0 


1 


HDEI 4b 

# audio ies listened to (succe^(^) 

□ 

0 


1 


HDE15 

satisfaction with quality of audo files 

0 

0 


4 


HDE16a 

#of uses of moping functions (attempted] 

3 

3 


2 


HDE16b 

#of uses of mapping functions (successfU) 

3 

3 


2 


HDE17 

satisfaction with mapping functions 

3 

2 


4 


HDE18 

satisfaction with SA fools 

□ 

3 


5 


HDE19a 

#of uses of stylus (attenriptec^ 

50 

7 


3 


HDE19b 

#of uses of stylus (successfJ] 

10 

7 


3 


HDE20 

satisfaction with stylus 

1 

4 


4 


HDE21a 

# of uses of buttons (atterrpted) 

0 

0 


0 


HDE21b 

# of uses of buttons (succ^sM) 

□ 

0 


0 


HDE22 

satisfaction with buttons 

□ 

0 


0 


HDE23 

satisfaction with one harxled operation 

□ 

1 


3 


HDE24 

satisfaction with removing device to use it 

0 

3 


0 


HDE25 

satisfaction with stowing device 

0 

3 


0 


HDE26 

abilily to use peisdid weepc^n 

0 

1 


3 


HDE27 

ability to read the screen 

□ 

5 


4 


HDE2S 

overall satis^ction with ulietiity 

1 

4 


5 


HDE29 

overall satisfaction with ruggedness 

□ 

0 


5 


HDE30 

overaJI satisfaction with system 

1 

4 


5 


HDE31 

impression of eased learning and use 

5 

5 


5 


HDE32 

free comments 

* 



A 
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No Device 



Participant Number 

0507 

□517 

□518 

TES1 

have cell phone (yes/no) 

□ 

1 

1 

TES2 

talk on phone (hours) 

□ 

6 

5 

TES3 

use text messaging (hours) 

0 

1 

0 

TES4 

customize cdl phono (hours) 

0 

1 

0 

TES5 

play games on cell phone (hours) 

0 

0 

0 

TES6 

phone numbers stored in cetl phone 

0 

40 

20 

TES7 

PDA use 

2 

1 

1 

TES8 

PDA stylus use 

4 

1 

1 

TES9 

video game play (hours) 

5 

8 

3 

TES10 

online video games (yes/no) 

1 

0 

0 

TES11 

home computer video games (yes/no) 

1 

0 

0 

TES12 

video game system video gaimes (yes/no) 

0 

1 

1 

TES13 

M favorite ^mes 

* 

A 

* 

TES14 

has work computer for own use{yes/no) 

0 

0 

1 

TES15 

has shared work computer (yes/no) 

1 

1 

0 

TES16 

almosi never use computer at work (yes/no) 

1 

1 

0 

TES17 

use computer in living quarters (yes/no) 

0 

0 

0 

TESie 

use public computers (yes/no) 

1 

1 

0 

TES19 

use several] programs a I same lime (yes/no) 

1 

1 

1 

TES201 

TES202 

TES203 

TES204 

TES205 

computer probem: a. stop 
computer probem: b read manusd 
compute probem: c. read hdp 
computer probem: d ask for help 
computer probem: e. keep trying 

1 

1 

5 

4 

3 

2 

1 


Participant Number 

No Device 

0507 0517 

0518 

TES21 

TES22 

use a computer 
*use email (hrs per week) 

35 

4 

100 

TES23 

'use email attachments (hrs per week) 

5 

2 

20 

TES24 

'use online chat cr IM (hrs per week) 

□ 

1 

0 

TES25 

'use internet to find information [hrs per week) 

7 

7 

50 

TES26 

'shop online (hrs per MONTH) 

0 

4 

2 

TES27 

'use online banking [hrs par MONTH) 

2 

2 

0 

TES28 

trust cornputers (yes/no) 

1 

1 

0 

TES29 

experience with radio 

1 

1 

3 

TES30 

experience with GPS 

□ 

1 

4 

TES31 

experience with topographical map 

2 

2 

5 

TES32 

experience with military compass 

3 

2 

5 

TES33 

experience with FBCB2 or BFT 

0 

1 

0 

TES34 

experience with Tacticomp 

0 

1 

0 

TES35 

experience with CDA or other RPDA 

□ 

1 

0 

TES36 

ago 

IS 

22 

35 

TES37 

gendsr 

M 

M 

M 

TES38 

payyade 

E2 

E4 

E5 

TES39 

branch of service 

USA 

USMC 

USA 

TES40 

years enlisted service 

1 

2 

9 

TES41 

years commissioned service 

0 

0 

0 

TES42 

designator. MOS, cr rating 

1ET 

2600 

11B2P 

TES43 

level; of education (1-5) 

2 

3 

2 

TES44 

current assignment 

* 

A 

* 

TES45a 

special schcds: Ranger 

0 

0 

0 

TES45b 

special sc hods: Pathfinder 

0 

0 

0 

TES45C 

special schods: EIB 

□ 

0 

0 

TES45d 

special schods: Air Assault 

0 

0 

0 

TES45e 

special schods: NOO Course 

0 

0 

1 

TES45f 

special schods: SNOO Course 

□ 

0 

0 

TES45g 

special schods: Corrputer Training 

□ 

0 

0 

TES45h 

special schods: HALO 

□ 

0 

0 
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SURVEY DATA: AGGREGATE SUMMARY OF RESPONSES 


Takagi Thesis Survey Aggregate Results 


BHIHW = 80% or more 
■ = 50% to 80% 

Eval Category 

Overall 

rfa 

no 

0 

yes 

1 

2 

3 

4 

5 

Total 

TES 

1 havecell ptione (yes/ro) 

36% 






14 

TES 

2 talk on phone [hours) 

median= 

5.00 

mean= 

4.69 

sd= 

409 

14 

TES 

3 use text messaging (hours) 

median= 


mean= 

0.77 

sd= 

0.93 

14 

TES 

4 customize cei 1 phone (hours) 

me()ian= 

0.20 

mean= 

0.48 

sd= 

0.51 

14 

TES 

5 play ganr»es on cell phone (hours) 

median^ 


meanF 

0.3S 

sd= 

065 

14 

TES 

6 phone numbers stored in cell phorte 

median^ 

30.00 

meariF 

33.46 

sd= 

32 24 

14 

TES 

7 PDA use 







14 

TES 

8 PDA slylus use 




43% 

7% 


14 

TES 

9 video game play (hours) ^ 

median= 

5.00 

mean= 

7.92 

sd= 

13.02 

14 

TES 

10 oriine video games (yes/no) 

EUL 

43% 





14 


TES 

TES 

TES 


11 h ome computer video games (yes/ro) 

12 video game system video games (yes/ro) 


43% 

29% 


14 

14 

13 


TES 

14 

has work computer for own use (yes/no) 


43% 





14 

TES 

15 

has shared work computer (yes/no) 


m 





14 

TES 

16 

almost never use computer at work [yes/no) 


31% 





13 

TES 

17 

use computer in living quarters (yes/no) 

46% 1 

54% 





13 

TES 

18 

use public computers (yes/no) 

31% 1 

69% 





13 

TES 

19 

use several programs at same time (yes/no) 

8% 1 

92*4 





13 

TES 

2G.1 

corrpiler problem: a. stop 



9% 


36% 

Ea 

11 

TES 

202 

corrpjter problem: b. read manuaJ 




27% 

27% 

45% 

11 

TES 

203 

computer problem: c. read help 


9% 

45% 

45% 



11 

TES 

204 

computer problem: d. ask for help 


8% 

33% 

25% 

33% 


12 

TES 

205 

computer problem: e. keep trying 

1 

86% 

14% 




14 


21 

use a computer 

median^ 

4.00 

meanF 

4.00 

sd= 

#DIV/0! 

1 

TES 

22 

use email [hrs per week) 

median= 

20.00 

mean= 

58.23 

sd= 

75.33 

14 

TES 

23 

use email attachments (hrs per week) 

median= 

5.00 

mean= 

18.92 

sd= 

30.15 

14 

TES 

24 

use online chat or tM (hrs per week) 

median= 


mean= 

79.15 

sd= 

276.71 

14 

TES 

25 

use internet to find information (hrs per week) 

median^ 

15.00 

meanF 

23.00 

sd= 

23.63 

14 

TES 

26 

shop online (hrs per MONTH) 

median^ 

2.00 

meanF 

2.15 

sd= 

1.82 

14 

TES 

27 

use online banking [ht^ per MONTH) 

median^ 

2.00 

meanF 

3.54 

sd= 

565 

14 

TES 

28 

trust compulersvirf person aJ info (yes/no) 

14% 1 

86% 





14 

TES 

29 

experience with radio 

14% 


7% 

7% 

14% 


14 

TES 

30 

experience with GPS 

36% 

43% 



14% 

7% 

14 

TES 

31 

e)Theriencewith topographical map 


21% 

36% 

21% 

7% 

14% 

14 

TES 

32 

e)Therieftce with military compass 


14% 

36% 

21% 

14% 

14% 

14 

TES 

33 

experience with FBCB2 or BFT 


14% 



7% 


14 

TES 

34 

experience with Tacdcomp 


21% 





14 

TES 

35 

e)^ience with COA or other RPDA 


14% 




7% 

14 
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Takagi Thesis Survey Aggregate Results 


= 80% or more Overall 


EVSl 

= 50% to 80% 

Category 

n/a 

no yes 

0 1 

2 

3 

4 

5 

Tot^ 

TES 

36 

age 

median^ 20.00 

meariF 

23.08 

sd= 

6.55 

14 

TES 

37 

gender 







14 

TES 

38 

pay grade 







14 

TES 

39 

bra rtch of service 







14 

TES 

40 

years enlisted service 

median= TOO 

meanp 

162 

sd= 

226 

14 

TES 

41 

years commissioned servcle 

median^ 

meanp 

0.75 

sd= 

2.60 

14 

TES 

42 

d^gnatcr. wos, or ralirtg 







14 

TES 

43 

level ofeducalion [1-5} 



36% 

43% 

21% 


14 

TES 

44 

current assignmeni 







11 

TES 

45a 

special schools: Ranger 






14 

TES 

45b 

special schools: Pathfinder 







14 

TES 

45c 

special schools: EIB 







14 

TES 

45d 

special schools: Air Assault 


H 7% 





14 

TES 

45e 

special sc hool s: NCO Course 


H 14% 





14 

TES 

45f 

special sc hod s: SNCO Cou rae 







14 

TES 

45g 

special sc hod s: Compu ter Tnai ni ng 







14 

TES 

45h 

special sc hods: HALO 







14 
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Takagi Thesis Survey Aggregate Results 


Eval 


= 80% or more 
= 50% to 80% 

Category 


Overall 

n/a 

no 

D 


yes 

1 


Tot^ 


HDE 

1 I 

HDE 

2 

HDE 

3 

HDE 

4 I 

HDE 

5a ; 

HDE 

5b i 

HDE 

e ! 

HDE 

7a i 

HDE 

7b i 

HDE 

8 I 

HDE 

9 I 

HDE 

10 

HDE 

11 ! 

HDE 

12a 

HDE 

12b 

HDE 

13 

HDE 

14a 

HDE 

14b 

HDE 

15 

HDE 

16a 1 

HDE 

16b 1 

HDE 

17 I 

HDE 

18 I 

HDE 

19a i 

HDE 

19b i 

HDE 

20 ! 

HDE 

21a 1 

HDE 

21b i 

HDE 

22 I 

HDE 

23 I 

HDE 

24 ! 

HDE 

25 ! 

HDE 

26 I 

HDE 

27 i 

HDE 

28 I 

HDE 

29 I 

HDE 

30 I 

HDE 

31 1 

HDE 

32 1 


Device used this trial 
Device used pr&viousfy 
location 

date 

description of scenario 
description of features 


satisfaction v 
# toKl messages received (attempted) 


satisfaction witri [exl messages received 
satisfaction with comm and network 
1D satisfaction with reiiability of comm and network 
satisfaction with si images 


35% 


35% 


29% 



8 % 

8 % 

8 % 8 % 
0.50 mean= 
050 meaoF 



0.50 

0.50 


meanp 

meanp 


83% 

2.17 

2.17 

2.25 

2.25 


25% 

42% 

sd= 

sd= 

8 % 

sd= 

sd= 

8 % 


17 

17 

0 

0 

0 

0 


median^ 

4.00 

meanF 

3.67 

sd= 

1.67 

12 

median^ 

3.50 

meanp 

3.58 

sd= 

1.68 

12 


8% 



Ka 

25% 

12 

median= 

3.00 

mean= 

3.33 

sd= 

1.50 

12 

median= 

3.00 

mean= 

3.33 

sd= 

1.50 

12 


e% 

2.88 

288 

42% 

2.99 

2.99 

42% 


12 

12 

12 

12 

12 

12 

12 

12 

12 

12 


# of uses of mapping functions (attempted) 

median= 

300 

mean= 

3.50 

sd= 

1.00 

12 

# of uses of mapping functions (successfii) 

median= 

3.00 

mean= 

3.50 

sd= 

1.00 

12 

satisfaction with mapping funcBons 



17% 

17% 1 

Eia 

S% 

12 

satisfaction with SA tods 

33% 



33% 

17% 

17% 

12 

# of uses of stylus (attempted) 

median^ 

10.00 

meanp 

18.08 

sd= 

21.79 

12 

# of uses of stylus (successfiJ) 

me<lian= 

10.00 

meanp 

14.75 

sd= 

19.39 

12 

satisfaction with st^us 


17% 


3% ] 

E9 

25% 

12 

# of uses of buttons (attempted) 

median= 


mean= 

0.08 

sd= 

029 

12 

# of uses of buttons (successfuH) _ 

median^ 


meanp 

0.08 

sd= 

029 

12 

satisfaction with buttons 

92“/. 



8% 



12 

satisfaction with one handed operation 

1 42% 

17% 

17% 

17% 


S% 

12 

satisfaction with removing device to use it 

75% 


8% 

17% 



12 

satisfaction with stowing device 

75% 



25% 



12 

ability to use personal weapon 

[~33% 

25% 

33% 

3% 



12 

ability to read the screen 





17% 

17% 

12 

overall satisfaction with reliability 

1 8% 

8% 

8% 


33% 

42% 

12 

overall satisfaction iMlh ruggedness 




17% 

8% 

25% 

12 

overall satisfaction with system 


8% 

8% 

1 

50% 

33% 

12 

impression of ease of learning and use 




3% 

17% 1 

75% 1 

12 
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Takagi Thesis Survey Aggregate Results 


= 80% or more 
■ifflM ^50%toS0% 


RPDA-D 

r/a 

no 


Eval 


Category 


D 


yes 

1 


2 


3 


4 


5 Tot^ 


TES 

1 have cell phone (yes/no) 

40% 1 

60*/^ 





5 

TES 

2 talk or) phono [hours] 

median= 

5.00 

meanp 

4.75 

sd= 

3.69 

5 

TES 

3 use text messaging [hours] 

median= 

150 

meanp 

125 

sd= 

0.96 

5 

TES 

4 customize cd 1 phone {hours) 

median= 

0.60 

meanp 

0.55 

sd= 

0.53 

5 

TES 

5 play ganrKs on cell phone [hours] 

median^ 

0.50 

meanp 

0.75 

sd= 

0.96 

5 

TES 

6 phene numbers stored in cell phone 

median^ 

57.50 

meanp 

53.75 

sd= 

4347 

5 

TES 

7 PDA use 


20% 





5 

TES 

8 PDA siyus use 


20% 


eo*4 



5 

TES 

9 video game play (hours] 

median= 

4.50 

mearF 

4.00 

sd= 

356 

5 

TES 

10 ort i ne vi deo games (yes/no) 

mm 

40% 





5 

TES 

11 h ome computer video games (yes/no) 

40% 1 






5 

TES 

12 video game system video games (yes/no) 

40% 






5 

TES 

13 list favorite games 







4 

TES 

14 has work computer for own use(yes/noJ 


40% 





5 

TES 

15 has shared work computer [yes/no) 

60% 

40% 





5 

TES 

16 a) most never use computer at work [yes/no] 

80*4 

20% 





5 

TES 

17 u se CO mpu ter i n [ivi ng quarters (yes/no) 


100*4 





5 

TES 

18 u se pu bl ic computers (yes/no) 


40% 





5 

TES 

19 use several programs at same time (yes/no) 

1 






5 

TES 

20.1 corrpi ter probl em: a. stop 



20% 


20% 1 


5 

TES 

20.2 compu ter proU em: b. read ma nu aJ 




40% 

20% 

40% 

5 

TES 

20.3 cornpu ter proU em: c. read hel p 



1 60% 

40% 



5 

TES 

20 4 computer proldem: d. ask for help 



20% 

20% 



5 

TES 

20 5 computer proUem: e. keep trying 

1 

100*4 





5 


21 use a computer 

median^ 

SHUMI 

meanp 

#DIW01 

sd= 

#DIV/0! 

0 

TES 

22 use email [hrs per week] 

median= 

110.00 

meanp 

106.25 

sd= 

106 43 

5 

TES 

23 use email attachmerrts (hrs per week) 

median= 

26 00 

mean= 

38.25 

sd= 

47.09 

5 

TES 

24 use online chat or IM [hrs per week) 

median= 

0.50 

mean= 

250.25 

sd= 

499 83 

5 

TES 

25 u se i ntern et b Jirtd I nformation [ hrs per week) 

median^ 

32.50 

meanp 

29.00 

sd= 

24.91 

5 

TES 

26 shop online [hrs per MONTH) 

median^ 

1.50 

meanp 

2.00 

sd= 

2.16 

5 

TES 

27 use online bank!ng (hrs per MONTH] 

median^ 

150 

meanp 

5.75 

sd= 

954 

5 

TES 

28 trust computers irtrf personal info [yes/no] 

20% 1 

80*4 





5 

TES 

29 experiencewithradio 


40% 

20% 


40% 


5 

TES 

30 experiencewithGPS 

1 

60*4 



20% 

20% 

5 

TES 

31 experi ence with topog raphicai map 


20% 

20% 

20% 

20% 

20% 

5 

TES 

32 experi ence with mal itary compass 



40% 


40% 

20% 

5 

TES 

33 experi ence with FBCB2 or BFT 

SO*/. 




20% 


5 

TES 

34 experi erree with Tacdcomp 

100% 






5 

TES 

35 experi erree with COA or ot her RPD A 

80*4 





20% 

5 
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Takagi Thesis Survey Aggregate Results 


= 80% or more RP D A-D 



50% to 80% 

r/a 








no yes 






EVSl 

Category 

0 1 

2 

3 

4 

5 

Tot^ 

TES 

36 age 

median^ 19.00 

meariF 

21.75 

sd= 

6.18 

5 

TES 

37 gender 






5 

TES 

38 pay grade 






5 

TES 

39 branch of service 






5 

TES 

40 years enlisted service 

median= TOO 

meanp 

0.75 

sd= 

0.50 

5 

TES 

41 years commissioned servcle 

median^ 

meanp 

2.25 

sd= 

450 

5 

TES 

42 designator, wos, or raiirig 






5 

TES 

43 level of educaBon [1-5} 


40% 

40% 

20% 


5 

TES 

44 current assignmenl 






5 

TES 

45a special schools: Ranger 






5 

TES 

45b special schools: Pathfinder 






5 

TES 

45c special schools: EIB 






5 

TES 

45d special schools: Air Assault 

20% 





5 

TES 

45e special sc hool s: NCO Course 

20% 





5 

TES 

45f special sc hool s: SNCO Cou rae 

BMW 





5 

TES 

45g special sc hool s: Compu ter Tnai ni ng 

100% 





5 

TES 

45 h special sc hool s: HALO 

100%^^^^ 





5 
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Eval 


= 80% or more 
= 50% to 80% 

Category 


RPDA-D 

r/a 

no 

D 


yes 

1 


Tot^ 


HDE 

1 I 

HDE 

2 

HDE 

3 

HDE 

4 I 

HDE 

5a 1 

HDE 

5b 1 

HDE 

e ! 

HDE 

7a i 

HDE 

7b i 

HDE 

8 1 

HDE 

9 1 

HDE 

10 I 

HDE 

11 ! 

HDE 

12a i 

HDE 

12b ^ 

HDE 

13 1 

HDE 

14a ^ 

HDE 

14b ^ 

HDE 

15 ! 

HDE 

16a ^ 

HDE 

16b ^ 

HDE 

17 1 

HDE 

18 1 

HDE 

19a t 

HDE 

19b i 

HDE 

20 ! 

HDE 

21a ^ 

HDE 

21b t 

HDE 

22 I 

HDE 

23 I 

HDE 

24 ! 

HDE 

25 ! 

HDE 

26 I 

HDE 

27 i 

HDE 

28 I 

HDE 

29 I 

HDE 

30 I 

HDE 

31 i 

HDE 

32 1 


Device used this trial 
Device used pr&viousfy 
location 

date 

description of scenario 
description of features 


satisfaction v 
# toKl messages received (attempted) 


satisfaction witri [exl messages received 
satisfaction with comm and network 
1D satisfaction with reliability of comm and network 
satisfaction with si images 


satisfaction with quafily of moving video 


median^ 4.00 

median^ 3.00 


median= 3.00 

median= 3.00 


meanp 

meanp 


mean= 

mean= 


mean= 

meaoF 


meanp 

meanp 


4.00 

3.S0 



2.40 

2.40 


satisfaction with quality of audo fries 





40% 

5 

# of uses of mapping functions (attempted) 

median= 3.00 

mean= 

3.80 

sd= 

1.10 

5 

# of uses of mapping functions (successfii) 

median= 3.00 

mean= 

3.80 

sd= 

1.10 

5 

satisfaction with mapping functions 





20% 

5 

satisfaction with SA tods 

40% 


40% 

20% 


5 

# of uses of stylus (attempted) 

median^ 10.00 

meanp 

11.40 

sd= 

498 

5 

# of uses of stylus (successfiJ) 

median= 10.00 

meanp 

11.40 

sd= 

498 

5 

satisfaction with st^us 



20% 

■39 

20% 

5 

# of uses of buttons (attempted) 

median= 

mean= 

- 

5d= 

- 

5 

# of uses of buttons (successfiri) 

median^ 

meanp 

- 

scl= 

- 

5 

satisfaction with buttons 






5 

satisfaction withono handed operation 


20% 




5 

satisfaction with removing device to use it 






5 

satisfaction with stowing device 

100% 





5 

ability to use personal weapon 


40% 




5 

ability to read the screen 

100% 





5 

overall satisfaction with reliability 

20% 



E9 

20% 

5 

overall satisfaction iMlh ruggedness 

40% 


40% 

20% 


5 

overall satisfaction with system 




80% 1 

20% 

5 

impression of ease of learning and use 



20% 

40% 

40% 

5 
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Eval 


= 80% or more 
= 50% to 80% 

Category 



Tot^ 


TES 

1 

TES 

2 

TES 

3 

TES 

4 

TES 

5 

TES 

6 

TES 

7 

TES 

8 

TES 

9 

TES 

1G 

TES 

11 

TES 

12 

TES 

13 

TES 

14 

TES 

15 

TES 

16 

TES 

17 

TES 

18 

TES 

19 

TES 

20. 

TES 

20. 

TES 

20. 

TES 

20 

TES 

20 

21 

TES 

22 

TES 

23 

TES 

24 

TES 

25 

TES 

26 

TES 

27 

TES 

28 

TES 

29 

TES 

30 

TES 

31 

TES 

32 

TES 

33 

TES 

34 

TES 

35 


have cell phone (yes/no) 
talk on phone [hours) 
use text messaging {hours) 
customize cdl phone {hours) 
play games on cell phono (hours) 
phono numbers stored in cell phono 
PDA use 
PDA slylus use 
video game play (hours) 


use a computer 


use online banking (hts per MONTH) 


experience with topographical map 


67*/* 1 





6 

5.00 

mean= 

5.17 

sd= 

5.19 

6 


mean= 

0.67 

sd= 

1.03 

6 

0.50 

meanp 

0.50 

sd= 

0.55 

6 


meanp 

0.33 

sd= 

0.52 

6 

25.00 

meariF 

26.67 

sd= 

26 58 

6 

m 

33% 

33% 



6 

6 

4.50 

mean= 

11.83 

sd= 

1902 

6 


17% 

83*4 1 





6 

medjan= 

^UM! 

meanp 

#DIV/0l 

sd= 

#DIV/0! 

6 

5n 

50% 1 





6 

59% 

50*4 





6 

80*4 

r 20% 





5 


40% 





5 


100*4 





5 

20% 

8[f4 





5 




1 

60*4 

1 40% 

5 




20% 

20% 

(MU 

5 


20% 

40% 

40% 



5 


17% 

33% 

33% 

17% 


6 


MSB 

33% 




6 

median^ 

4.00 

meanp 

4.00 

sd= 

#DIV/0! 

1 

median= 

15.00 

meanp 

32.17 

sd= 

5347 

6 

median= 

300 

mean= 

11.00 

sd= 

19.49 

6 

median= 

250 

mean= 

4.50 

sd= 

596 

6 

median^ 

13.50 

meanp 

19.S3 

sd= 

26 04 

6 

median^ 

3.00 

meanp 

2.33 

sd= 

1.86 

6 

median^ 

2.00 

meanp 

3.17 

sd= 

382 

6 


100*4 





6 

33% ISa 





6 


33% 





6 


33% 

33% 

33% 



6 


33% 

33% 

33% 



6 

83*4 

1 17% 





6 

1 67% 

33% 





6 

83*4 

1 17% 





6 
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Takagi Thesis Survey Aggregate Results 


- 8M or more 
■lt!liia-SO%toBO% 

RPDA-A 

1 r/a 









no 

yes 






Eval 


Category 

0 

1 

2 

3 

4 

5 

Tot^ 

TES 

36 

age 

median^ 

20.50 

meariF 

23.00 

sd= 

6.69 

6 

TES 

37 

gender 







6 

TES 

38 

pay grade 

median= 

#JUM! 

meanp 

#DIW[)! 

sd= 

#DIV/0! 

6 

TES 

39 

branch of service 

median= 

muw. 

meanp 

#DIW[)! 

sd= 

#DIV/0! 

6 

TES 

40 

years enlisted service 

median= 

TOO 

meanp 

TOO 

sd= 

- 

6 

TES 

41 

years commissioned servcle 

median^ 


meanp 

- 

sd= 

. 

6 

TES 

42 

designator, wos, or raiirig 

median^ 

90.00 

meanp 

90.00 

sd= 

#DIV/0! 

6 

TES 

43 

level ofeducalion [1-5} 



17% 


33% 


6 

TES 

44 

current assignmenl 







3 

TES 

45a 

special schools: Ranger 

100% 






6 

TES 

45b 

special schools: Pathfinder 

100% 






6 

TES 

45c 

special schools: EIB 

100% 






6 

TES 

45d 

special schools: Air Assault 

100% 






6 

TES 

45e 

special schools: NCO Course 

100% 






6 

TES 

45f 

special schools: SNCO Course 

100% 






6 

TES 

TES 

45g 

45h 

special schools: Computer Training 
special schools: HALO 

100% 

100% 






6 

6 


0 

0 

0 
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Takagi Thesis Survey Aggregate Results 


= 80% or more 
= 50% to 80% 


Eval 


Category 



Device used this trial 

Device used previousfy 

HDE 

1 

location 

HDE 

2 

date 

HDE 

3 

description of scenario 

HDE 

4 

description of features 

HDE 

5a 

# leKt messages sent (attempted) 

HDE 

5b 

#lexl rnessages sent (successfii) 

HDE 

e 

satisfaction with text messages sent 

HDE 

7a 

# teKl messages received (attempted) 

HDE 

7b 

#teKl messages received [successful) 

HDE 

8 

satisfaction with [exl messages received 

HDE 

9 

satisfaction with comm and network 

HDE 

10 

satisfaction with reliability of comm and network 

HDE 

11 

satisfaction with si images 

HDE 

12a 

# moving videos viewed (attempted) 

HDE 

12b 

# moving videos vi^Ared (successtul) 

HDE 

13 

satisfaction with quafily of moving video 

HDE 

14a 

# audio files listened to (aitempfed) 

HDE 

14b 

# audio files listened to (successful) 

HDE 

15 

satisfaction with quality of audo fries 

HDE 

16a 

# of uses of mapping functions (attempted) 

HDE 

16b 

# of uses of mapping functions (successfii) 

HDE 

17 

satisfaction with mapping functions 

HDE 

18 

satisfaction with SA tods 

HDE 

19a 

# of uses of stylus (attempted) 

HDE 

19b 

# of uses of stylus (successfiJ) 

HDE 

20 

satisfaction with st^us 

HDE 

21a 

# of uses of bu tton s (attempted) 

HDE 

21b 

# of uses of buttons (succeesfil) ^ 

HDE 

22 

satisfaction with buttons 

HDE 

23 

satisfaction withono handed operation 

HDE 

24 

satisfaction with removing device to use it 

HDE 

25 

satisfaction with stowing device 

HDE 

26 

ability to use personal weapon 

HDE 

27 

ability to read the screen 

HDE 

28 

overall satisfaction with reliabilily 

HDE 

29 

overall satisfaction iMlh ruggedness 

HDE 

30 

overall satisfaction with system 

HDE 

31 

impression of ease of learning and use 

HDE 

32 

free comments | 


meanF 

meanF 


mearF 

mearF 



25% 


25% 25% 


25% 



Tot^ 

6 

6 

0 

0 

0 

0 


4.00 

4.00 

sd= 

sd= 

1.41 4 

1.41 4 

1 

100% 

1 ^ 

4.25 

4.25 

sd= 

sd= 

25% 

096 4 

0.96 4 

4 


100% 



50% 

1 25% 4 

26% 

50% 

4 

3.25 

sd= 

320 4 

3.25 

sd= 

3.20 4 

Wfl-rM 4 

3.50 

sd= 

3.51 4 

3.50 

sd= 

3.51 4 

4 

3.75 

sd= 

0.96 4 

3.75 

sd= 

0.96 4 

25% 

Ka 

1 4 

25% 

25% 

25% 4 

25.00 

sd= 

3336 4 

25.00 

sd= 

33.36 4 


25% 

4 

0.25 

sd= 

0.50 4 

0.25 

sd= 

0.50 4 
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= 80% or more 
= 50% to 80% 


Eval 


Category 



Device used this trial 

Device used pr&viousfy 

HDE 

1 

location 

HDE 

2 

date 

HDE 

3 

description of scenario 

HDE 

4 

description of features 

HDE 

5a 

# leKt messages sent (attempted) 

HDE 

5b 

#lexl messages sent (successfii) 

HDE 

e 

satisfaction with text messages sent 

HDE 

7a 

# teKl messages received (attempted) 

HDE 

7b 

#teKl messages received [successful) 

HDE 

8 

satisfaction with [exl messages received 

HDE 

9 

satisfaction with comm and network 

HDE 

10 

satisfaction with reliability of comm and network 

HDE 

11 

satisfaction with si images 

HDE 

12a 

# moving videos viewed (attempted) 

HDE 

12b 

# moving videos vi^Ared (successtul) 

HDE 

13 

satisfaction with quafily of moving video 

HDE 

14a 

# audio files listened Id (aitempfed) 

HDE 

14b 

# audi 0 fil es lis tened Id (successful] 

HDE 

15 

satisfaction with quality of audo fries 

HDE 

16a 

# of uses of mapping functions (attempted) 

HDE 

16b 

# of uses of mapping functions (successfii) 

HDE 

17 

satisfaction with mapping functions 

HDE 

18 

satisfaction with SA tods 

HDE 

19a 

# of uses of stylus (attempted) 

HDE 

19b 

# of uses of stylus (successfiJ) 

HDE 

20 

satisfaction with st^us 

HDE 

21a 

# of uses of buttons (attempted) 

HDE 

21b 

# of uses of buttons (succeesfull) _ 

HDE 

22 

satisfaction with buttons 

HDE 

23 

satisfaction withono handed operation 

HDE 

24 

satisfaction with removing device to use it 

HDE 

25 

satisfaction with stowing device 

HDE 

26 

ability to use personal weapon 

HDE 

27 

ability to read the screen 

HDE 

28 

overall satisfaction with reliabilily 

HDE 

29 

overall satisfaction iMlh ruggedness 

HDE 

30 

overall satisfaction with system 

HDE 

31 

impression of ease of learning and use 

HDE 

32 

free comments 


RPDA-B 

n/a 

no 

D 


yes 

1 


Tot^ 


median^ 

1,00 

meanp 

2.67 

sd= 

2.89 

3 

median^ 

1,00 

meanp 

2.67 

sd= 

2.89 

3 


33% 



Ka 


3 

median= 

100 

mean= 

2.67 

sd= 

289 

3 

median= 

1,00 

mean= 

2.67 

sd= 

289 

3 





33% 1 

WSl'M 

3 



33% 




3 







3 



33% 

83% 

33% 


3 

median= 


mean= 

0.33 

sd= 

058 

3 

median= 


mean= 

0.33 

sd= 

058 

3 

sa 




33% 


3 

median^ 


meanp 

0.33 

sd= 

0.58 

3 

median^ 


meanp 

0.33 

sd= 

0.58 

3 

wm 




33% 


3 

median= 

300 

mean= 

2.67 

sd= 

058 

3 

median= 

3.00 

mean= 

2.67 

sd= 

0.58 

3 



33% 

33% 

33% 


3 

33% 



33% 


33% 

3 

median^ 

7,00 

meanp 

20.00 

sd= 

26 06 

3 

median= 

7,00 

meanp 

6.67 

sd= 

3.51 

3 


33% 



mm 


3 

median= 

- 

mean= 

- 

sd= 

- 

3 

median^ 


meanp 

- 

sd= 

- 

3 

100% 






3 

I 33% 

33% 


33% 



3 

67 */< 



33% 



3 

67 */. 



33% 



3 

33% 

33% 


33% 



3 

33% 




33% 

33% 

3 


33% 



33% 

33% 

3 






33% 

3 


33% 



33% 

33% 

3 





1 

100% 1 

3 

median= 

SMUM! 

mean= 

#DIW[)! 

sd= 

#DIV/0! 

3 
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5. USABILITY TEST DATA 

The observations recorded using Noldus Observer were summarized 
using Elementary Statistics, Total Durations, and then exported to Microsoft 
Excel. The tables below show the data in Excel, formatted for clarity. The Scored 
By column indicates the coder who scored that video. When both coders scored 
the same file, the REV (reviewed) column was created by averaging the two 
observations. If the two observations were off by more than ten seconds, the 
original video was reviewed to see if one coder had made an error or 
misinterpreted the scoring criteria. In that case, the REV column contains the 
more accurate score based on the later review. 


Participant ID 

501 

502 

503 

503 

503 

504 

504 

504 

505 

505 

505 

506 

501 

502 

Scored By 

KB 

KT 

KB 

KT 

REV 

KB 

KT 

REV 

KB 

KT 

REV 

HT 

KB 

KB 

Device Tested 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

B 

B 

Time in Service 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Pay Grade 

E3 

E4 

E2 

E2 

E2 

E3 

E3 

E3 

E2 

E2 

E2 

E4 

E3 

E4 

Easy to Learn 

5 

5 

0 

0 

0 

0 

5 

5 

5 

5 

5 

5 

5 

1 

Overall Satisfaction 

2 

5 

0 

0 

0 

□ 

5 

5 

5 

5 

5 

4 

4 

5 

on 1c commchk 

0 

□ 

0 

0 

0 

□ 

□ 

0 

0 

□ 

0 

0 

0 

0 

on 2c plot rte 

□ 

□ 

0 

0 

0 

□ 

□ 

0 

0 

0 

0 

0 

0 

0 

on 3c read grid 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 4c brg rng 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on Sc send rr^g 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 6c rev msg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 7c read back 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on la commchk 

38 

74 

36 

43 

36 

0 

0 

0 

35 

45 

40 

26 

0 

0 

on 2a plot rte 

□ 

□ 

0 

0 

0 

□ 

□ 

0 

0 

0 

0 

0 

0 

0 

on 3a read grid 

130 

289 

104 

221 

104 

39 

42 

405 

12 

44 

12 

82 

162 

79 

on 4a brg rng 

18 

79 

29 

38 

38 

□ 

□ 

0 

43 

116 

43 

111 

0 

0 

on 5a send msg 

154 

247 

194 

197 

196 

64 

72 

68 

131 

85 

131 

121 

79 

60 

on 6a rev msg 

117 

15 

26 

28 

27 

0 

0 

0 

16 

18 

17 

13 

0 

0 

on 7a read back 

8 

9 

9 

9 

9 

4 

8 

6 

7 

6 

65 

25 

0 

0 
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Participant ID 

502 

502 

503 

503 

503 

504 

504 

504 

505 

505 

505 

506 

513 

513 

Scored By 

KT 

REV 

KB 

KT 

REV 

KB 

KT 

REV 

KB 

KT 

REV 

KT 

KB 

KT 

Device Tested 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

D 

D 

Time in Service 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Pay Grade 

E4 

E4 

E2 

E2 

E2 

E3 

E3 

E3 

E2 

E2 

E2 

E4 

E2 

E2 

Easy to Learn 

1 

1 

0 

0 

0 

□ 

5 

5 

5 

5 

5 

0 

3 

3 

Overall Satisfaction 

5 

5 

0 

0 

0 

0 

5 

5 

5 

5 

5 

0 

4 

4 

on tc commchk 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 2c plot rte 

□ 

0 

0 

0 

0 

□ 

□ 

0 

0 

0 

0 

0 

0 

0 

on 3o read grid 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 4c brg rng 

a 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on Sc send msg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 6c rev msg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 7c read back 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 1a commchk 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

22 

22 

on 2a plot rte 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

56 

54 

on 3a read grid 

106 

79 

52 

133 

52 

108 

131 

108 

60 

78 

78 

45 

11 

9 

on 4a brg mg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

25 

25 

on 5a send msg 

65 

62 5 

57 

56 

66.5 

aa 

84 

86 

110 

111 

111 

57 

92 

92 

on Ga rev msg 

□ 

□ 

0 

0 

0 

□ 

7 

0 

0 

0 

0 

0 

19 

25 

on 7a read back 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

5 

6 
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Participant ID 

513 

514 

514 

514 

515 

527 

528 

501 

502 

503 

503 

503 

504 

505 

Scored By 

REV 

KB 

KT 

REV 

KB 

KB 

KB 

KB 

KT 

KB 

KT 

REV 

KT 

KB 

Device Tested 

D 

D 

D 

D 

D 

D 

D 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Time in Service 

1 

1 

1 

1 

1 

9 

9 

1 

1 

1 

1 

1 

1 

1 

Pay Grade 

E2 

E2 

E2 

E2 

E3 

EB 

03 

E3 

E4 

E2 

E2 

E2 

E3 

E2 

Easy to Learn 

3 

4 

4 

4 

4 

5 

5 

5 

0 

0 

0 

0 

5 

0 

Overall Satisfaction 

4 

4 

4 

4 

4 

5 

4 

4 

0 

0 

0 

0 

5 

0 

Oil 1 g commchk 

0 

0 

0 

0 

0 

0 

0 

5 

0 

13 

19 

16 

0 

13 

on 2c plot rte 

0 

□ 

0 

0 

0 

0 

0 

5Q2 

264 

314 

322 

318 

197 

290 

on 3o read grid 

0 

0 

0 

0 

0 

0 

0 

100 

40 

27 

19 

23 

97 

31 

on 4c brg rng 

0 

0 

0 

0 

0 

0 

0 

89 

78 

45 

53 

49 

0 

106 

on Sc send msg 

0 

0 

0 

0 

0 

0 

0 

5 

15 

23 

28 

25 5 

0 

15 

on 6c rev msg 

0 

0 

0 

0 

0 

0 

0 

52 

30 

32 

22 

32 

0 

44 

on 7c read back 

0 

0 

0 

0 

0 

0 

0 

3 

14 

33 

32 

325 

0 

7 

on 1a commchk 

22 

19 

25 

22 

2 

1 

1 

0 

0 

0 

0 

0 

0 

0 

on 2a plot rte 

55 

118 

46 

46 

92 

46 

42 

0 

0 

0 

0 

0 

0 

0 

on 3a read grid 

10 

17 

14 

15.5 

12 

12 

13 

0 

0 

0 

0 

0 

0 

0 

on 4a brg mg 

25 

27 

38 

325 

8 

12 

18 

0 

0 

0 

0 

0 

0 

0 

on 5a send msg 

92 

63 

62 

625 

78 

93 

78 

0 

0 

0 

0 

0 

0 

0 

on Ga rev msg 

22 

8 

3 

5.5 

12 

16 

4 

0 

0 

0 

0 

0 

0 

0 

on 7a read back 

5.5 

54 

55 

54.5 

3 

20 

5 

0 

0 

0 

0 

0 

0 

0 

Participant ID 

505 

505 

506 

513 

513 

513 

514 

514 

514 

515 

527 

527 

527 

528 

Scored By 

KT 

REV 

KT 

K8 

KT 

REV 

KB 

KT 

REV 

KB 

KB 

KT 

REV 

KB 

Device Tested 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Time in Service 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

9 

9 

9 

9 

Pay Grade 

E2 

E2 

E4 

E2 

E2 

E2 

E2 

E2 

E2 

E3 

E6 

E6 

E6 

03 

Easy to Learn 

0 

0 

0 

3 

0 

0 

4 

0 

0 

0 

0 

5 

5 

5 

Overall Satisfaction 

□ 

0 

0 

4 

0 

0 

4 

0 

0 

0 

0 

5 

5 

4 

on 1c commchk 

15 

145 

14 

29 

32 

30.5 

9 

14 

11.5 

15 

29 

10 

10 

7 

on 2c plot rte 

309 

290 

321 

305 

309 

307 

273 

258 

258 

286 

211 

139 

189 

231 

on 3c read grid 

56 

31 

23 

30 

34 

32 

12 

12 

12 

31 

1 

4 

2.5 

5 

on 4c brg rng 

119 

113 

111 

80 

80 

80 

88 

83 

85.5 

84 

38 

37 

375 

38 

on 5c send msg 

29 

15 

23 

15 

16 

15.5 

36 

37 

365 

15 

61 

66 

635 

18 

on 6c rev msg 

79 

44 

51 

16 

26 

21 

30 

43 

43 

18 

8 

29 

29 

27 

on 7c read back 

5 

6 

18 

12 

12 

12 

12 

10 

11 

5 

1 

5 

3 

0 

on la commchk 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 2a plot rte 

□ 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 3a read grid 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 4a brg mg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on Sa send msg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 6a rev msg 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

on 7a read back 

□ 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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APPENDIX L 


MORS-TISDALE PRESENTATION 


This thesis was the winning entry in the Military Operations Research 
Society’s (MORS) - Tisdale Graduate Research Award competition. The 
following presentation was presented to the faculty of the NFS Operations 
Research Department for the competition on September 6, 2005. 



Put this slide up immediately and leave up for 
introduction. Go to next slide automatically when 
speaker gets to that point. 

'Good afternoon, I am Captain Koichi Takagi and I 
am honored to be here in front of you. ” 

[NEXT SLIDE] 


Bn (kground 

Applied WarFigliler Ergonnmks Center 
Appro nrli 
Ctise Sludy 
Condudons 

Que^lion and Answer period 

Slide # 2 

"I will present an overview of my thesis, and briefly 
demonstrate some of the capabilities of our 
laboratory. I have a lot of backup material on hand, 
so at the end, we can go into more detail about any 
of the topics" 

[PAUSE - NEXT SLIDE] 


technical note: 

AWE Blue: Red 0, Green 114, Blue 188 
Presentation font: TW Cen MT Condensed 
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Slide #3 

"None of this would have been possible without the 
support of our research sponsors and the patience 
of my dear family and friends." 

[NEXT SLIDE] 


Slide #4 

"I also gratefully acknowledge the many students 
and faculty that have made this last seven-hundred 

and ninety-one days, seven hours, and_minutes 

bearable, enriching, and meaningful. " 

[NEXT SLIDE] 


[0:35] 


[* from start of refresher qtr, 8 July 2003, to 1500 on 
Sep 6;-)] 
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Background 


Slide # 5 

[CLICK VIDEO TO PLAY, AND THEN CLOSE 
WINDOWS MEDIA PLAYER COMPLETELY] 

"End users bear the burden of poor design. Usability 
testing must involve representative participants 
under realistic conditions. There is a human factors 
gap in the T&E process. This video shows British 
Royal Marines landing in Basra using equipment 
that may not have been tested by the right people 
under the right conditions." 

Various military agencies are currently fielding a 
number of devices known collectively as 
Ruggedized Personal Digital Assistants, or RPDAs, 


KruA u^hfQrking I dh^lMyi and c-annsTDi 


Ruggediied Personal Digital AssislanlsfRPDAs) 

iiCti 


Slide #6 


including these, which we have been testing 
within the Tactical Network Topology project 
RPDAs have unique hum an factors considerations. 


Civilian PDAs are not designed to be used while the 
operator 


[NEXT SLIDE] 


[NEXT SLIDE] 



Military RPDAhuman factors consider a No ns 
L SimultunGDUs Ejefnandi on artcntion 

2. Other individuol equipment and weapons 

3. Environmental extremes 


Slide # 7 



Military RPDA human factors considorations 
1. Simultaneaui demands on aTlenrion 
?. Oihe r indivi duOl equi pme nt and weapons 
3. E nvi ro nmental extre mes 


Slide #8 


"... simultaneously attends to mental and physical 
stimuli," 


".. wears or employs other equipment and weapons, 
or..." 


[NEXT SLIDE] 


[NEXT SLIDE] 
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Military RPDA human factors considerations 

1. Simultaneous demands on attention 

2. Other individual equipment and weapons 

3. Environmental extremes 


Slide # 9 

"... operates under environmental extremes. 

The Special Operations Command desired a human 
factors evaluation of the RPDAs they have fielded in 
Iraq and Afghanistan." 

[NEXT SLIDE] 


Interested Partners 
Marine Corps Wartighting Lab 
SOCOM (various components) 

US Army Special Operations Command 

US Army Rapid Equipping Force 

US Army PEC Soldier 

US Navy Human Performance Center 

US Forest Service 

NPS MOVES Institute 

NPS Computer Science and ITM departments 

Noldus Information Technology Slide# 11 

"We developed a state-of-the-art behavioral 
observation laboratory that has generated serious 
interest from a number of different agencies and is 
the basis for much future research at NPS. The 
developer of the software is using our lab as a 
model on their website." 

[NEXT SLIDE] 


Mission Statement 
To design and establish a laboratory 
and a methodology 
for scientifically evaluating 
the human factors considerations 
of combat equipment used by military personnel 
under both controlled laboratory 
and realistic field conditions. 

Slide# 10 

"At the time, no DoD facility existed that was 
equipped to specifically test military individual 
equipment in the manner of software usability 
testing. 

My research interest was to perform scientifically 
rigorous human factors evaluations that went 
beyond subjective observations and survey data. 
There was a need within the DoD for a laboratory 
and a reliable research methodology to do just that." 

[NEXT SLIDE] 


Observing a user navigating 
the NPS Python web page 



Slide# 12 

Here in the lab, an observer watches the 
participant’s computer screen broadcast to his own, 
while watching through a one-way mirror and 
recording everything to DVD files for later analysis." 

[NEXT SLIDE] 
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52’’ sJereo plosma display 



Slide #13 


TartiiaE scenario 

[USHC Conib or Decision Range #32} 


Slide #14 



"Your handouts give more detail about the lab 
equipment. 

Aside from testing PC-based software, we can 
immerse the participants" 

[NEXT SLIDE] 


...in a multimedia tactical scenario..." 

[PLAY video clip: 14 participant watching screen] 

"... the order of march is a HMMWV with a .50 cal, 
two trucks, and a HMMWV in the rear." 


[NEXT SLIDE IMMEDIATELY WHEN VIDEO ENDS] 


Two fixed^ robotic digital yidao cameras 


Portable and wireless digital video ramoras 



We record everything they do from front and side 
views..." 


[NEXT SLIDE] 


"and can record their actions using a handheld 
device." 

[NEXT SLIDE] 
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Mobile Device Camera demo 



Slide# 17 

[ switch to live feed from Mobile Device Camera: S-video input on 
table ] 

[begin demonstration of actual equipment] 

"Up on stage here with me we have a Marine in typical combat dress. 

I have built battery packs for the equipment so that a researcher can 
follow a user in the field, recording how they operate the equipment 
and the context. We are seeing exactly what he does on the screen of 
his RPDA. All the necessary equipment fits into three waterproof 
transit cases. An actual field study could validate the results obtained 
from a controlled laboratory experiment." 


Noldus Observer software 



Slide# 18 

"We bring all these digital videos back into the 
Noldus Observer software, where observed 
behaviors are manually coded into data files." 

[NEXT SLIDE] 


[NEXT SLIDE] 


[pass out lab manuals] 


Video control station hardware 



There are a lot of hardware and software 
components to the system to install," 

[NEXT SLIDE] 


Professional recording equipment 



...operate, and maintain. So that this lab will 
continue to be a valuable resource to the 
department," 


[NEXT SLIDE] 

[including a digital video mixer, event coding 
keyboard, digital camera controller] 

[Video mix monitors, mic and speaker amps, the 
wireless camera and microphone receivers, and the 
computer workstation and its associated software.] 
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Slide #21 


Users guide 





Slide #22 


"...I’ve written a Users Guide, which distills 
hundreds of pages of documentation " 


" and usability references into a focused, stand¬ 
alone tutorial." 

[NEXT SLIDE] 


[NEXT SLIDE] 


Approach for usability testing 


Approach 


]. Definition of need 

2 . User identification 

3. Focos groops & interviews 

4. Laboratory and field testing 

5 . Pre- and post-operation sorveys 
A. Analysis and final report 


Slide # 23 

[4:30] 

"I also wanted to pave the way for future research by 
devising and validating..." 

[NEXT SLIDE] 


Slide #24 

"a start-to-finish methodology for performing a 
usability study of military individual equipment. I 
used qualitative and quantitative means to dissect 
the problem of human factors usability testing." 

[NEXT SLIDE] 

(My thesis details my experimental design, data 
collection method, analytical procedure, lessons 
learned and my recommendations for future 
researchers.) 
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Case Study 

Case Study 

Three devices: 

RPDA-A RPDA-B RPDA-D 

Slide #25 

kj Hi 

Slide # 26 

"To validate this laboratory and procedure, 1 
conducted a small case study comparing three 
different RPDAs." 

"The case study is a proof-of-concept and was not 
expected to be conclusive. This study is continuing, 
to achieve the degree of test significance that 

SOCOM desires" 

[NEXT SLIDE] 

[NEXT SLIDE] 

Participants 

Should be representative of intended population 

Must be available as test participants 

Between-sub|ects design due ta availability at 
devices and participants (never all at ance) 

Pre-operation survey results 

64°/ have a cell phone (5 hr/wk median talk time) 

50°/o have never operated PDA before 

54°/ have a computer at home 
median age 20 yrs old 

84°/ directly from boot camp 

Slide #27 

Slide # 28 

"Soldiers and Marines at DLI brought to NPS in 
small groups. {A between-subjects experimental 
design will usually be appropriate because neither 
the different devices nor the participants will be 
available for the whole study }" 

"The pre-operation survey has 45 questions that 
attempt to capture the participant's technology 
experience and their military background." 

[5:00] 

[NEXT SLIDE] 

[NEXT SLIDE] 
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Tactical scenario 



"The participants went through a 30-minute tactical 
scenario, during which they performed..." 


Tactical tasks 

1. Perform 0 communications check 

2. Ploto route 

3. Determine three mop coordinotes 

4. Determine the hearing and range fro m one point to 
another 

3. Send o message 

6. Receive o message 

7. Retrieve o previous message from storage 

Slide #30 

"...various communications and navigation tasks. 
The scenario drove them to perform these tasks 
using conventional means" 


[NEXT SLIDE] 


Tactical scenario script 


tM Patrol Bmo. 


[NEXT SLIDE] 



Slide #32 


"...(such as with a paper map), to provide a baseline 
measure of their abilities. Later, they performed 
similar tasks using the RPDA." 

[Pause, NEXT SLIDE] 


[play 33 correct rpda read grid] 

"If the quantitative results are not conclusive, we 
may still be able to make recommendations based 
on our observations in the lab and the field." 


[NEXT SLIDE] 
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Interface errors 



Slide #33 

[play 33 clicking interface error] 

"In this segment, it took 23 screen taps to close the 
dialog box 

A great advantage of digital video collection is that 
you can go back to the original data to interrogate 
outliers" 


Tactical reaction to the scenario, pt. I 



Slide # 34 

"We saw a broad range of behaviors during the 
experiment" 

[play 35 tactical reaction] 

[NEXT SLIDE AS SOON AS VIDEO FINISHES] 


[NEXT SLIDE] 


Tactical reaction to the scenario, pt. II 



Slide #35 

[play 36 nonchalant reaction] 

"some of which are correlated to their survey 
responses" 

[NEXT SLIDE] 


Conventional mapping, pt. I (Marine) 



Slide # 36 

[play 37 conv mapping while standing] 

"We saw a range of techniques employed. For 
example, some groups took different tactical 
postures" 

[NEXT SLIDE AS SOON AS VIDEO FINISHES] 
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Conventional mapping, pt. II (Soldier) 



Slide #37 

[play 38 conv mapping spread] 

" than others. One group invariably removed their 
weapons when mapping. 

Some assessments were subjective..." 

[NEXT SLIDE AS SOON AS VIDEO FINISHES] 


Participant off-task due to confusing instructions 



Slide # 39 


Unresponsive interface 



Slide #38 

[play 39 non responsive] 

"We tried to differentiate between time-consuming 
tasks, hardware issues, and users misunderstanding 
the task" 

[NEXT SLIDE] 


Physical errors due to device configuration 



Slide #40 


[play 40 confused] 

"For a subjective measurement, how would you 
know when a tester is confused?" 

[MAKE SURE AUDIENCE HEARS HIM SAY, "I'M 
CONFUSED" - REPLAY IF NECESSARY] 

[NEXT SLIDE] 


[play 41 fumbling with cords] 

"This device is tethered to networking equipment 
and large batteries in the backpack, which might not 
be an optimal configuration." 

[NEXT SLIDE] 
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Coded data files 


Plot of coded tasks 


118.93 on Ic ooinn diK 
128.03 off Ic com chk 

147.26 on 2c plot rte, /* often reshuffling rifle; 

snitched pencils; kneels for task 
387.46 off 2c plot rte 

449.96 ndsteke, /* incorrect brg/mg values for both 
545.76 tactical posture 
563.43 on 5c send msg 

565.13 admin posture 
582.00 on 3c read grid 
594.66 off 3c real grid 

596.90 slip, /* reads 4 cfmrd grid; tac;tical stress 

613.70 on 5c send msg, /* h<^ nsg info r^lai^ed 

636.90 off 5c send msg 

734.56 on 6c rev msg, /* kneels to wite info 

771.70 on 4c brg rng, /* kneels 

859.13 mistake, incmrrect distance 


Slide #41 



Slide # 42 


"Researchers code the videos into raw data files like 
these, that describe every observed event." 

[NEXT SLIDE] 


Inter-rater reliability 


Confusion matrix 

t1 t2 t3 t4 t5 t6 t7 Error Total A 

t1 22.63 . 0.33 22.96 

t2 - 54.73 . 0 54.73 

t3 - • 9.71 - - - - 0 9.71 

t4 - - ■ 25.1 - - - 0.14 25.24 

15 - ... 9206 - 0 92.06 

16 . 19.53 - 5.8 25.33 

17 . 5.5 0.94 6.44 

Error 0 1.63 165 0 0.17 0 0 - 3.46 

TotB22.e3 56.36 11.36 25.1 92 23 19.53 5.5 7.21 239.92 


"One way to look at that data is through this time- 
step plot, which shows what the participant was 
doing throughout the scenario." 

[NEXT SLIDE] 


Inter-rater reliability 

Confusion matrix 

11 12 13 14 15 16 17 Error Total A 

11 22.63 . 033 22.96 

12 - 54.73 . 0 54.73 

13 • • 9.71 .... 0 9.71 

14 • - - 25.1 - ■ - 0.14 25.24 

15 - ... 92.06 - - 0 92.06 

16 - .... 1953 . 5S 25.33 

17 . 5.6 094 6.44 

Error 0 1.63 1.65 0 0.17 0 0 - 3.46 

Tot.022.63 56.36 1136 25.1 92.23 1953 5.5 721 239.92 


Task time assessments by two different 
researchers on the same video file 

Slide #43 

"To verify that the measurements are objectively 
coded, we look at inter-rater reliability." 

[NEXT SLIDE] 


Slide # 44 

"We compare the task times assigned by a different 
researcher on the vertical and horizontal axes and 
find very good agreement in this observation" 

[NEXT SLIDE] 
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Inter-rater reliability 


Confusion matrix 

t1 t2 t3 t4 t5 t6 t7 Error Total A 

11 22.63 ------ 0.33 22.96 

12 - 54.73 ----- 0 54.73 

13 - - 9.71 - - - - 0 9.71 

14 - - - 25.1 - - - 0.14 25.24 

15 - - - - 92.06 - - 0 92.06 

16 - - - - - 19.53 - 5.8 25.33 

17 - ----- 5.5 0.94 6.44 

Error 0 1.63 1.65 0 0.17 0 0 - 3.45 

Tot.B22.63 56.36 11.36 25.1 92.23 19.53 5.5 7.21 239.92 


Duration of agreements: 229.3 sec 

Duratian of disagreements: lO.titi sec 
Index afcancordance: 0.96 

Cahen'sK: 0.94 Slide # 45 

"There are standard statistics we can use to ensure 
we are using objective measurement criteria." 

[NEXT SLIDE] 

[The index of concordance is just the proportion of 
agreements to total observed time. 

Cohen's kappa is a measure of the proportion of 
agreement that exceeds that expected by chance: 

K = (observed proportion - proportion expected by 
chance ) / (1 - proportion expected by chance ) 


Analytical method 

Adjust measurements for participant effects 
Analyze task times for a device effect 
Use parametric techniques for a richer model 
Use non-parametric techniques for insurance against 
poor assumptions and small data sets 


Slide #47 

"Quantifying individual differences between 
participants reduces the noise in the measurements, 
and reduces our required sample size. 

With a properly constructed experiment, we can 
justify quantitative analysis of human factors data." 

[NEXT SLIDE] 


Conclusions 


Slide #46 

"There are a few key points I considered in my case 
study, and propose for future studies" 

[NEXT SLIDE] 


Participant effect 

Between-participant variability is greater than 
between-device variability 

Finding factors that explain participant variability: 

Pre-operation surveys 

Conventional tasks to establish baselines 

Slide #48 

"Testable measures of participant effect, like the 
baseline tasks, are preferable to surveys, which are 
prone to a variety of errors." 

[NEXT SLIDE] 
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Paranietric analysis 

Parametric onalyris 

ANOVAmMlel: 

AN QVA model: 

> |-5_uiv-jirw(l-.5--ilfiv,dA|-d.-T|iElA3 

Vi \ hIhiI Y d.iHima i^mirifttnn^n ri3itnr-w.ls far :^ni;X f X-nd. 
f 1, TM-► i ^giiii by Tifcfly ^D^ttiocl 


CBtlaatB Std.Errox loMor ■ound ITpfiar HdukL 
m-H i^.7 -A_d1 7dl.O 

i ^0««9 l-iSt 

m-D Hl.fi 7B.i l.%9 nO.D 

a-c -la.o i2.7 -im.oo «3.1 

Slide # 49 

Slide #50 

"Here's a simple example of analyzing this data. In 

"and conclude that using RDA-D is in fact faster than 

task 5, we see a difference between devices," 

RPDA-A. We also see that device B is better than A 

[NEXT SLIDE] 

at a slightly lower confidence level." 


[NEXT SLIDE] 

Nan parametric analysis 

Subje [live analysis 

Kruskal-wallis Tesl 


> dmT) 

liig.hlighr video dips 

p ^.ILW. - Qj&ui 


JLE.iuhiJ.-'HmlllH KMlLl-L'iviriHnii mHlmllv {S - OjH)! : 

B ri Bcaer ilwi A, and □ rs betKr man A ftaw as AHWA) 

Pd^I operalian survey results 

l$-Pms CW* Uf narti-cefH(nn»nt CHUttpl] 



Subieci mailer expert review 

Slide # 51 

Slide #52 

We arrive at the same conclusions non¬ 

"On the other hand, some users did voice strong 

para metrically. However, when adjusting for 

opinions about the devices in their post-operation 

participant effects, we did not find a statistically 

surveys. My research has shown that perception can 

significant difference between the devices in this 

influence the adoption of innovations more strongly 

study." 

than quantitative performance. 


[NEXT SLIDE] 

[NEXT SLIDE] 
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Slide # 53 

"My thesis fills the human factors gap in the T&E 
process. There are Special Forces operators and 
Marines in Iraq and Afghanistan using all of these 
devices today . My thesis results will be applied by 
SOCOM to improving current and future RPDAs. 

This laboratory will make the Naval Postgraduate 
School the premiere military individual equipment 
testing site in the country." 

[BE PREPARED TO SWITCH TO THESIS IN 
WORD] 


Great things about my thesis experience 

Created a research question exactly in line with my 
professional and academic interests. 

Faced few constraints in devising my research plan. 

Was actively involved in collecting my own data. 

Had complete freedom in designing and establishing 
a laboratory specifically for my research. 

Slide # 54 

"Some great things about my thesis experience are 
that I was able to pursue a research question exactly 
in line with my professional and academic interests, I 
faced few constraints in devising my research plan, I 
was actively involved in collecting my own data, and 
I had complete freedom in designing and 
establishing a laboratory specifically for my 
research. A few of the drawbacks were that I... " 

[NEXT SLIDE] 


"The bulk of my thesis effort was in designing and 
testing the laboratory and structure, and I'd be 

hannv/ tn nn intn mnro Htatcailc ohm it it I ran olcn 


Drawbacks of my thesis experience 
Had to come up with my own research questions. 

Had to come up with my own plan. 

Had to get my own data. 

Had to build my own lab... 


Slide # 55 

"...had to come up with my own research questions, 
my own research plan, I had to get my own data, 
and I had to build my own lab..." 

[NEXT SLIDE] 
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